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Abstract 

In  1998,  the  Aerospace  Communications  and  Information  Expertise  (ACE) 
program  was  implemented  to  provide  a  common  operational  foundation  for  new  Air 
Eorce  Communications  and  Information  officers.  Training’s  crucial  role  in  providing  Air 
Eorce  effectiveness  and  efficiency  in  the  officer  corps  is  demonstrated  by  the  formal 
training  courses  new  officers  are  required  to  attend  for  instruction  in  their  jobs.  The 
importance  of  training,  and  subsequent  training  evaluation,  is  evident  for  two  significant 
reasons:  the  skills  required  by  Air  Eorce  Communications  and  Information  officers  and 
the  amount  of  investment  in  training.  Investment  in  training  includes  money,  time, 
equipment,  and  any  other  significant  factor  that  contributes  to  training  and  education  of 
personnel  in  order  for  them  to  acquire  the  knowledge,  skills,  and  attitudes  needed  to 
perform  their  job.  While  training  and  training  evaluation  research  is  progressing,  there  is 
still  a  lack  of  training  evaluation  as  well  as  training  effectiveness  methodologies  at  this 
time.  By  developing  and  testing  an  appropriate  training  effectiveness  model  that  will  aid 
in  determining  whether  or  not  training  is  effective;  this  research  seeks  to  aid  in  increasing 
effectiveness  of  BCOT. 
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AN  ANALYSIS  OF  THE  AIR  FORCE  BASIC  COMMUNICATIONS  OFFICER 


TRAINING  COURSE:  THE  IMPACT  OF  TRAINEE  AND  ORGANIZATION 
CHARACTERISTICS  ON  TRAINING  EFFECTIVENESS 


I.  Introduction 


Overview 

This  chapter  discusses  the  need  for  training  Air  Foree  Communieations  and 
Information  (C&I)  offieers  and  the  importanee  of  evaluating  the  effeetiveness  of  the 
Basic  Communications  Officer  Training  (BOOT)  eourse  as  the  first  formal  training 
reeeived  by  Air  Foree  C&I  Offieers  in  their  professional  training  eontinuum.  A  brief 
deseription  of  the  BOOT  eourse  as  well  as  a  theoretieal  argument  supporting  the  need  for 
training  and  training  evaluation  is  developed  by  linking  issues  of  training  effeetiveness 
and  training  performanee  to  BOOT  training.  Next,  the  problem  statement  and  researeh 
focus  are  presented.  Finally,  generalizability  of  this  researeh  is  discussed. 

Background 

In  1998,  the  Aerospaee  Communieations  and  Information  Expertise  (ACE) 
program  was  implemented  to  provide  a  eommon  operational  foundation  for  new  Air 
Foree  Communieations  and  Information  offieers.  Basie  Communieations  Offieer 
Training  (BCOT)  is  required  within  six  months  of  new  aecessions  being  assigned  to  their 
ACE  tour  (Department  of  the  Air  Foree,  1998).  Enough  time  has  passed  sinee  the 
ineeption  of  ACE  to  decide  if  BCOT  is  effeetive  in  training  ACE  aecessions. 
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Training’s  crucial  role  in  providing  Air  Force  effectiveness  and  efficiency  in  the 
officer  corps  is  demonstrated  by  the  formal  training  courses  new  officers  are  required  to 
attend  for  instruetion  in  their  jobs  (Alliger  &  Tannenbaum,  1996).  As  mentioned  earlier, 
new  C&I  officers  attend  BCOT.  The  intent  of  the  BCOT  course  is  to  provide  junior  C&I 
offieers  a  eommon  foundation  of  Air  Force  eommunications  skills  (Department  of  the  Air 
Foree,  2002).  Eleven  bloeks  of  instruction  ranging  from  networking  to  deployable 
communieations  systems  are  taught  in  the  BCOT  course  (Department  of  the  Air  Force, 
2002).  This  curriculum  supports  the  career  plan  for  C&I  officers  developed  at  the  1998 
Utilization  and  Training  Workshop  (Department  of  the  Air  Force,  1998). 

The  importanee  of  training,  and  subsequent  training  evaluation,  is  evident  for  two 
signifieant  reasons:  the  skills  required  by  Air  Foree  C&I  officers  and  the  amount  of 
investment  in  training.  Skill  requirements  for  the  C&I  offieer  career  field  are  listed  in  Air 
Foree  Manual  36-2105,  Officer  Classifieations  (Department  of  the  Air  Foree,  2001). 
Training  is  used  to  solve  numerous  problems  such  as  developing  new  skills,  knowledge, 
understanding,  and  attitudes  (Johnson,  1976).  The  need  to  evaluate  training  is  identified 
in  the  Guidebook  for  Air  Force  Instructors  which  outlines  the  Instructional  Systems 
Development  (ISD)  model  used  extensively  in  all  Air  Foree  training  programs 
(Department  of  the  Air  Force,  1998).  Unfortunately,  training  evaluation  in  the  military  is 
rarely  completed  because  of  misconceptions  sueh  as  the  environment  preventing 
evaluation  or  the  trainers  disregarding  evaluation  beeause  they  are  confident  the  training 
works  without  proof  (Salas,  Milham,  &  Bowers,  2003). 

Investment  in  training  includes  money,  time,  equipment,  and  any  other  significant 
factor  that  contributes  to  training  and  education  of  personnel  in  order  for  them  to  aequire 
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the  knowledge,  skills,  and  attitudes  needed  to  perform  their  job.  One  estimate  indieates 
the  amount  of  money  spent  on  individual  and  eollective  training  exceeds  $27.1  billion 
(Salas  et  ah,  2003).  The  federal  government  alone  is  estimated  to  spend  approximately 
$633  million  (Faerman  &  Ban,  1993).  Financial  commitments  toward  training  support 
developing  skills  to  meet  a  multitude  of  today’s  challenges  (Tracey  &  Tews,  1995).  With 
the  amount  of  money  and  other  resources  used  on  training  C&I  officers  to  develop  skills 
needed  to  perform  their  duties,  it  is  important  that  the  training  be  evaluated  to  ensure  C&I 
officers  are  learning  what  is  needed  for  performance  in  the  field.  It  also  stands  to  reason 
that  those  investing  in  training  would  want  to  know  whether  or  not  they  are  getting  a 
return  on  their  investment  (Parry,  1996). 

J.  Kevin  Ford  (1997)  observed  how  independent  literature  on  training  appeared 
during  the  1920s  and  remained  scant  until  the  1940s  when,  in  1949,  the  first  review  of 
training  literature  was  completed.  While  training  and  training  evaluation  research  is 
progressing,  there  is  still  a  lack  of  training  evaluation  as  well  as  training  effectiveness 
methodologies  at  this  time  (Salas  et  ah,  2003).  Thus,  the  question  remains,  how  is  this 
return  on  investment  measured?  In  other  words,  how  is  training  effectiveness  measured? 

Problem  Statement 

In  the  Air  Force,  numerous  parties  are  interested  in  whether  or  not  training  is 
effective.  Senior  leadership  desires  to  have  a  professional  work  force.  Initial  training  is 
the  first  step  in  developing  such  a  force.  Additionally,  immediate  supervisors  desire 
highly  competent  workers  enabling  immediate  missions  to  be  accomplished.  Further, 
training  support  personnel  (i.e.,  trainers  and  curriculum  builders)  need  to  know  if  the 
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current  training  program  is  effective  in  training  ACE  aceessions  for  their  ACE  tour  and 
beyond.  Einally,  trainees  need  to  feel  eonfident  the  training  provided  to  them  is  at  the 
level  needed  to  perform  their  respeetive  jobs.  By  developing  and  testing  an  appropriate 
training  effectiveness  model  that  will  aid  in  determining  whether  or  not  training  is 
effeetive,  this  researeh  seeks  to  aid  in  inereasing  effeetiveness  of  BCOT. 

Research  Focus 

Integrating  eonstruets  from  the  several  studies  and  the  Mathieu  and  Martineau 
(1997)  theorized  model,  the  researeh  foeus  for  this  study  is  to  investigate,  in  a  military 
setting,  the  relationships  between  trainee  eharacteristies,  organization  eharaeteristies,  and 
training  outcomes  as  posited  in  the  researeher’s  proposed  training  effeetiveness 
measurement  model  presented  in  chapter  two. 

Survey  data  eolleeted  by  the  researeher  from  BCOT  attendees  will  be  used  to 
assist  in  determining  the  proposed  relationships  identified  earlier.  The  results  of  this 
study  could  aid  training  support  personnel  at  BCOT  in  providing  the  most  effeetive 
training  to  BCOT  attendees.  In  turn,  ACE  aecessions  would  be  suffieiently  prepared  for 
their  ACE  tour  and  beyond.  The  specifie  researeh  questions  that  will  be  addressed  to 
examine  the  relationship  between  trainee  eharaeteristies,  organization  eharacteristies,  and 
training  outcomes  are: 

1 .  Based  on  the  literature,  which  constructs  are  appropriate  for  measuring  training 
effeetiveness? 

2.  What  is  the  relationship  between  trainee  eharacteristies  and  training  outcomes? 

3.  What  is  the  relationship  between  organization  eharacteristies  and  training 
outcomes? 


4 


This  research  posits  that  certain  trainee  characteristics  and  organization 
characteristics  (i.e.,  affective  organization  commitment,  task-related  self-efficacy, 
learning  self-efficacy,  training  motivation,  organization  support,  organization  constraints, 
and  opportunity  to  perform  skills  learned)  are  related  to  training  outcomes  as  measured 
by  the  trainee’s  performance,  reaction  to  training,  and  motivation  to  transfer  skills  learned 
to  the  job.  This  study  integrates  selected  models  to  examine  the  relationship  between 
training  and  organization  characteristics  and  immediate  training  outcomes  as  well  as 
possible  future  training  outcomes.  Furthermore,  training  performance  and  reaction  are 
expected  to  be  indicators  of  immediate  training  effectiveness,  while  motivation  to  transfer 
is  expected  to  be  indicative  of  future  job  performance  which  may  or  may  not  be  a  further 
indication  of  training  effectiveness. 

Relationships  between  these  constructs  and  training  effectiveness  may  provide 
insight  into  issues  that  may  be  hindering  training  effectiveness.  In  turn,  by  identifying 
possible  problems  and  potential  solutions  with  the  current  training  and  training  support, 
further  development  of  future  Air  Force  C&I  Officers  through  training  will  be  possible. 

Generalizability 

With  one  of  the  goals  of  training  being  the  ability  for  the  trainee  to  transfer 
learned  knowledge,  skills,  and  attitudes  to  the  work  environment,  measuring  training 
effectiveness  becomes  important  in  the  evaluation  of  any  training  program.  Apart  from 
measuring  task-related  self-efficacy  levels,  great  effort  has  been  taken  to  develop  and  test 
a  training  effectiveness  measure  that  may  be  used  for  any  training  program  in  any 
environment.  From  a  systems  perspective,  when  linking  the  individual  to  the 
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organization,  training  outcomes  should  bolster  organizational  goals.  From  a  high  level 
perspective,  it  logieally  follows  that  when  a  trainee  strives  to  learn  knowledge,  skills, 
behaviors,  and  attitudes  during  training,  the  organization  benefits  from  the  transfer  of  that 
learning  to  the  work  environment.  In  turn,  intent  for  the  trainee  to  transfer  such  learned 
objectives  could  be  an  indieation  of  effective  training.  While  the  focus  group  for  this 
study  was  a  military  professional  development  and  skills  training  course,  it  is  hoped  that 
future  research  will  further  the  reliability  and  validity  of  the  measurement  instrument 
developed  for  this  study  in  an  effort  to  develop  a  generalized  measure  for  training 
effectiveness  that  foeuses  on  the  appropriate  trainee  and  organization  characteristics. 

Summary 

This  chapter  introduced  the  proposed  researeh  to  determine  the  effeetiveness  of 
BCOT  training  by  measuring  identified  trainee  and  organization  characteristics.  Chapter 
two  will  discuss  the  literature  identified  to  support  sueh  training  evaluation  and  present 
the  proposed  research  model.  Chapter  three  will  discuss  the  methodology  used  to 
conduct  the  research  and  propose  in  depth  investigative  questions  to  answer  the  researeh 
questions  outlined  in  this  chapter.  Chapter  four  will  discuss  the  results  obtained  from  the 
researeh.  Finally,  a  discussion  of  conclusions  from  the  researeh  and  future  researeh  ideas 
will  be  presented  in  chapter  five. 
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II,  Literature  Review 


Overview 

This  chapter  reports  on  the  literature  concerning  training  evaluation,  trainee 
eharacteristies,  organization  eharaeteristics,  training  outeomes,  and  training  effectiveness. 
Previous  researeh  will  be  presented,  as  well  as  the  studies  used  to  examine  the 
relationship  between  trainee  characteristics,  organization  characteristics,  and  training 
outcomes.  Following  an  in  depth  review  of  the  research  literature,  a  theoretical 
framework  for  measuring  training  effectiveness  of  the  Basie  Communieations  Officer 
Training  (BOOT)  eourse  will  be  proposed.  The  utility  of  the  proposed  model  will  be 
geared  towards  supporting  course  development  for  BOOT  and  supporting  force 
development  of  the  Communications  and  Information  (C&I)  Officer  eareer  field. 

Training  Evaluation 

Kirkpatrick’s  Framework. 

Aceording  to  Bennett,  Alliger,  Eddy,  and  Tannenbaum  (2003),  “[tjraining 
evaluation  is  the  programmatie  process  whereby  the  outcomes  of  training  are  tracked  and 
analyzed  (p.  60)”.  Donald  Kirkpatriek’s  (1976)  4-stage  training  evaluation  model  has 
been  used  eontinuously  by  the  military  to  measure  training  effeetiveness. 

Kirkpatrick’s  model  uses  a  goal-based  evaluation  approaeh  to  measure  reaetion, 
learning,  behavior,  and  results.  Reaction  is  defined  by  Kirkpatrick  as  how  well  the 
trainee  liked  the  training  program.  Two  studies  (Alliger,  Tannenbaum,  Bennet  Jr., 

Traver,  &  Shetland,  1997;  Warr  &  Bunce,  1995)  discerned  factorially  distinct  forms  of 
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reaction.  Alliger  et  al.  distinguished  between  affective  (i.e.,  how  enjoyable  the  training 
was  for  the  trainee)  and  utility  (i.e.,  how  useful  the  training  was  as  judged  by  the  trainee) 
reaction,  finding  the  latter  to  be  better  associated  with  subsequent  job  performance.  Warr 
and  Bunce  identified  three  forms  of  reaction,  namely,  enjoyment  of  training,  perceived 
usefulness  of  training,  and  perceived  difficulty  of  training.  Reaction  to  training  is  the 
basic  or  lowest  level  of  training  evaluation. 

At  the  higher  levels  of  evaluation,  Kirkpatrick  defines  the  learning  level  of 
evaluation  as  the  knowledge  understood  and  retained  by  the  trainee.  The  behavior  level 
is  defined  by  Kirkpatrick  as  job  performance  after  completion  of  the  training.  Finally,  the 
results  level  is  defined  as  the  outcomes  that  appear  on  the  job  as  a  result  of  training. 

Expanding  Kirkpatrick’s  Framework. 

While  Kirkpatrick’s  model  provides  a  practitioner-friendly  method  for  evaluating 
training,  more  recent  studies  have  included  measurements  of  individual  and  organization 
characteristics  and  their  impact  on  training,  escalating  the  complexity  of  an  already 
intricate  task  (Noe,  1986;  Tannenbaum  &  Yukl,  1992).  Furthermore,  evaluating  training 
from  a  systems  perspective,  specifically  focusing  on  how  training  transfers  to  the 
organization,  is  another  recent  addition  to  training  evaluation  and  to  training  effectiveness 
measurement  (Mathieu,  Martineau,  &  Tannenbaum,  1993;  Tannenbaum  &  Yukl,  1992). 

Use  of  Kirkpatrick’s  model  as  a  first,  global  heuristic  for  training  evaluation  has 
worked  well  (Alliger  &  Janak,  1989).  However,  as  Watkins,  Leigh,  Foshay,  and 
Kaufman  (1998)  and  Alliger  and  Janak  (1989)  mention,  there  are  assumptions  associated 
with  Kirkpatrick’s  model  that  may  present  the  need  to  question  this  and  other  similar 
models.  Assumptions  of  Kirkpatrick’s  model  include  arranging  the  four  levels  of 
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evaluation  in  an  ascending  order  of  value;  causally  linking  the  levels;  and  positively 
correlating  the  levels  (Alliger  &  Janak,  1989).  Nullmeyer  and  Spiker  (2003)  further 
contend  that  rigid  adherence  to  Kirkpatrick’s  model  has  resulted  in  lost  opportunities  to 
measure  training  effectiveness.  Alliger  and  Tannenbaum  (1996)  concur  with  Nullmeyer 
and  Spiker  by  stating  Kirkpatrick’s  well-known  scheme  is  not  the  last  word  in  training 
criteria,  and  that,  if  taken  too  literally,  Kirkpatrick’s  model  will  hinder  evaluation  efforts. 

Numerous  research  efforts  have  suggested  ways  to  not  only  overcome  the 
assumptions  of  Kirkpatrick’s  model,  but  to  expand  the  model  to  include  other  equally 
important  areas  of  study.  Alliger  and  Janak  (1989)  suggests  expanded  measurement  at 
each  of  Kirkpatrick’s  levels  to  include  trainees,  peers,  subordinates,  and  supervisors  in  an 
effort  to  completely  capture  the  criterion  at  each  level.  Kraiger,  Ford,  and  Salas  (1993) 
note  that  Kirkpatrick,  among  others,  have  ignored  affectively  based  measures  such  as 
organizational  commitment  as  indicators  of  learning.  They  further  proposed  a  broader 
range  of  affectively  or  attitudinally  based  outcomes  for  measurement  that  may  infer 
learning  during  training.  Noe  (1986)  uses  Kirkpatrick’s  model  in  a  linear  fashion  where 
each  level  of  evaluation  affects  the  next  level  in  the  hierarchy.  However,  his  theoretical 
discussion  expands  the  research  to  include  important  motivational  and  situational  factors 
from  organizational  behavior  theory.  He  suggests  that  these  constructs  may  attenuate  or 
enhance  the  effectiveness  of  training.  Tannenbaum  and  Woods  (1992)  note  that 
expansion  of  the  Kirkpatrick  model  to  include  attitude  change  can  help  identify  trainee’s 
beliefs,  convictions,  and  attitudes  toward  training  and  possible  transfer  of  training  after 
course  completion.  In  their  meta-analysis  of  34  studies,  Alliger  et  al.  (1997)  developed 
an  augmented  framework  using  Kirkpatrick’s  taxonomy  as  a  guide.  They  expanded 
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reactions  to  include  affective  and  utility  reactions;  they  expanded  learning  to  include 
immediate  knowledge,  knowledge  retention,  and  behavior/skill  demonstration;  they 
renamed  behavior  to  transfer;  and  they  left  level  four  evaluation  (i.e.,  results)  the  same. 
Kraiger  et  al.  (1993)  viewed  training  outcomes  as  multidimensional,  meaning  changes 
may  occur  in  cognitive,  affective,  and  skill  capacities.  Further,  measurement  of  the 
effects  of  individual,  organizational,  and  training-related  factors  may  explain  why 
training  does  or  does  not  work  supporting  the  need  for  training  evaluation  to  measure 
training  effectiveness.  They  note  that  learning  outcomes  are  not  discrete  but  often 
interrelated  and  thus  changes  in  one  learning  outcome  may  involve  changes  in  another. 
Kraiger  et  al.  (1993)  further  discuss  examining  the  relationship  between  changes  in 
learning  outcomes  and  other  important  training  outcomes  to  advance  understanding  of 
training  evaluation  and  training  effectiveness.  Colquitt,  LePine,  and  Noe  (2000) 
reviewed  research  that  supported  examining  how  personal  characteristics  related  to 
training  effectiveness  providing  support  for  the  concept  that  what  the  trainee  brings  to 
training  is  important.  They  further  emphasized  the  importance  of  conducting  training 
studies  that  determine  whether  individual  and  situational  characteristics  explain  any 
incremental  variance  in  training  outcomes.  With  Colquitt  et  al.’s  (2000)  training 
motivation  meta-analysis,  the  necessity  to  integrate  their  work  with  earlier  research  on 
training  settings  and  methods  was  identified  to  uncover  other  intervening  mechanisms 
linking  individual  and  situational  characteristics  with  training  motivation  and  learning. 

Studies  have  also  begun  to  consider  the  interface  between  the  organization  and 
training  (Kozlowski  &  Salas,  1997).  The  incorporation  of  a  systems  perspective 
identifies  issues  that  must  be  addressed  to  ensure  training  contributes  to  the 
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organization’s  goals.  Furthermore,  preparing  individuals  for  training  and  eneouraging 
transfer  of  learned  skills  to  the  work  environment  requires  training  to  be  delivered  at  the 
appropriate  level  and  with  the  appropriate  organization  support.  Salas,  Cannon-Bowers, 
and  Kozlowski  (1997)  identify  that  training  theories  now  range  from  individual-level 
proeesses  to  organizational-level  systems.  They  contend  that  training  embodies  a 
complex  set  of  individual  and  organizational  variables  that  interact  dynamically  to 
produce  learning  outcomes  at  all  levels:  individuals,  groups,  and  organizations.  In  turn, 
training  effectiveness  has  also  expanded  to  incorporate  individual  and  organizational 
characteristics.  Ford  (1997)  contends  there  is  a  need  to  pay  more  attention  to  training  as 
part  of  the  organizational  context  to  include  examining  pre-training  and  post-training 
environments  and  how  they  impact  training  success.  With  the  broader  issues  involved  in 
understanding  training  effectiveness,  development  of  conceptual  models  are  also 
important  to  identify  factors  prior  to,  during,  and/or  following  training  that  may  impact 
learning,  retention,  and  transfer.  Evaluation  is  now  promoted  as  part  of  a  cyclic  process 
in  the  Instructional  Systems  Design  process  (Alliger  &  Tannenbaum,  1996).  Alliger  and 
Tannenbaum  support  an  iterative  process  which  includes  planning,  design, 
implementation,  and  evaluation  in  which  evaluation  should  be  a  part  of  each  step  of  the 
ISD  process,  not  just  a  singular  process  at  the  end  of  the  training.  Alliger  and 
Tannenbaum  define  training  effectiveness  evaluation  as  “the  determination  of  the  impact 
of  training  in  terms  of  some  dependent  measure  or  measures,  such  that  ‘impact’  means  a 
change  or  improvement  in  those  measures  (p.  6).”  They  go  on  to  state  such  measurement 
may  assess  declarative  or  procedural  knowledge  or  may  even  be  indicators  of  post¬ 
training  behavior  (i.e.  transfer  of  learning  to  the  job).  An  integrated  training  evaluation 
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approach  may  serve  as  early  warning  signs  for  trainers  of  trainee  or  eurriculum  problems 
preventing  wasted  training  effort.  Eseryel  (2002)  also  identifies  evaluation  as  an  integral 
part  of  the  instructional  design  model  that  may  help  determine  the  effectiveness  of  the 
instructional  intervention.  Eseryel  emphasizes  the  complexity  involved  in  evaluation 
with  regard  to  learning,  transfer,  and  organizational  impaet.  He  presented  six  general 
approaehes  to  edueational  evaluation:  goal-based,  goal-free,  responsive,  systems, 
professional  review,  and  quasi-legal.  As  stated  earlier,  Kirkpatrick’s  model  follows  the 
goal-based  evaluation  method.  And,  while  systems-based  models  may  be  better  for 
overall  context,  they  may  not  be  suffieient  in  representing  interaction  between  training 
design  and  training  evaluation.  Overall,  Eseryel  concludes  that  evaluation  is  eomplex 
and  not  always  well-structured  and  that  future  needs  include  a  eyclical  approach  to 
incorporating  training  evaluation  into  the  instructional  design  model  (Eseryel,  2002). 
Eurthermore,  Bell  and  Kerr  (1987)  contend  evaluation  is  needed  even  with  its  complexity 
and  that  lack  of  evaluation  may  lead  to  continuation  or  even  proliferation  of  ineffective 
training  programs. 

Table  1  summarizes  the  most  relevant  research  that  has  expanded  on 
Kirkpatrick’s  training  evaluation  model.  Overcoming  the  assumptions  of  Kirkpatrick’s 
model  and  further  expanding  training  evaluation  researeh  to  include  individual  and 
organizational  characteristics  as  supported  by  the  above  research  will  further  training 
effectiveness  researeh.  The  next  seetion  presents  the  trainee  characteristics,  organization 
characteristics,  and  training  outcomes  seleeted  for  researeh  in  this  study. 
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Table  1.  Summary  of  Research  Relevant  to  Expanding  Kirkpatrick’s  Framework 


Researcher(s) 

Trainee 

Characteristics 

Organization 

Characteristics 

Training  Outcomes 

Alliger  and  Janek  (1989) 

X 

X 

Alliger  and  Tannenbaum  (1996) 

X 

X 

X 

Cannon-Bowers,  Salas,  Tannenbaum,  and  Mathieu  (1995) 

X 

X 

Colquitt,  LePine,  and  Noe  (2000) 

X 

X 

X 

Eseryel  (2002) 

X 

X 

X 

Ford  (1997) 

X 

X 

Ford,  Quinones,  Sego  and  Sorra  (1992) 

X 

X 

X 

Frayne  and  Fatham  (1988) 

X 

X 

Kozlowski  and  Salas  (1997) 

X 

X 

Kraiger,  Ford,  and  Salas  (1993) 

X 

X 

X 

Mann  (1996) 

X 

X 

Mathieu,  Martineau,  and  Tannenbaum  (1993) 

X 

X 

Mathieu,  Tannenbaum,  and  Salas  (1992) 

X 

X 

X 

Noe  (1986) 

X 

X 

Noe  and  Sehmitt  (1986) 

X 

X 

X 

Pintrich,  Smith,  Garcia,  and  McKeachie  (1993) 

X 

X 

Ryman  and  Biersner  (1975) 

X 

X 

Salas,  Cannon-Bowers,  and  Kozlowski  (1997) 

X 

X 

X 

Tannenbaum  and  Woods  (1992) 

X 

X 

Tannenbaum  and  Yukl  (1992) 

X 

X 

X 

Tracey,  Hinkin,  Tannenbaum,  and  Mathieu  (1997) 

X 

X 

X 

Warr,  Allan,  and  Birdi  (1999) 

X 

X 

Warr  and  Bunce  (1995) 

X 

X 

Trainee  Characteristics 


Trainee  Attitudes. 

Affective  Organization  Commitment.  Organization  commitment  has  been 
linked  to  training  effectiveness  in  several  studies.  By  defining  organization  commitment 
as  how  much  an  individual  identifies  with  and  is  involved  with  an  organization, 
Tannenbaum,  Mathieu,  Salas,  and  Cannon-Bowers  (1991)  linked  organization 
commitment  to  training  effectiveness  by  studying  how  a  trainee’s  level  of  organization 
commitment  influenced  his  view  of  training  usefulness.  Tracey  et  al.  (1997)  agreed  that 
when  defined  in  this  manner;  organization  commitment  could  positively  influence  pre¬ 
training  self-efficacy,  in  turn,  influencing  training  effectiveness.  Tracey  and  Tews  (1995) 
suggested  that  if  an  individual  possesses  a  high  degree  of  organization  commitment,  the 
individual  may  in  turn  view  training  as  worthwhile  and  have  more  commitment  to 
learning  material  in  the  course.  Tracey  further  considered  that  organization  commitment 
may  affect  the  trainee’s  attitude  toward  work  and  have  an  affect  on  the  trainee’s 
preparation  for  and  application  of  training.  O’Connor,  Peters,  Pooyan,  Weekley,  Frank, 
and  Erenkrantz  (1984)  conducted  a  field  investigation  and  found  a  significant  association 
between  inhibiting  situational  constraints  and  turnover  (i.e.,  very  low  organization 
commitment)  at  all  managerial  levels  he  investigated.  Finally,  Colquitt  et  al.’s  (2000) 
review  of  research  suggested  that  higher  levels  of  organizational  commitment  may  cause 
the  trainee  to  view  training  as  useful  to  themselves  and  the  organization. 

Meyer,  Allen,  and  Gellatly  (1990)  contributed  further  to  the  organization 
commitment  construct  by  developing  scales  to  measure  two  distinct  views  of 
commitment,  that  is,  affective  and  continuance  commitment.  Allen  and  Meyer  (1990) 
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define  affective  organization  commitment  as  the  emotional  attachment  to,  identification 
with,  and  involvement  in  the  organization.  Their  research  supports  the  employee’s  need 
to  feel  comfortable  within  the  organization  and  competent  in  the  work-role.  Further, 
results  of  their  study  indicated  that  employees  who  did  feel  these  two  needs  were  met, 
expressed  greater  affective  attachment  to  the  organization.  Meyer  and  Allen  (1984) 
conducted  two  studies  that  posited  the  more  investment  the  trainee  makes  to  the 
organization  (e.g.,  completing  training  courses  for  more  job  competence),  the  higher  his 
affective  organization  commitment  will  be.  Given  the  literature  supporting  affective 
organization  commitment  and  considering  the  training  effectiveness  context  of  this 
research,  affective  commitment  (i.e.,  commitment  best  predicted  by  personal  competence 
and  positive  work  experiences)  is  the  more  pertinent  construct.  Therefore,  affective 
organization  commitment  was  measured  in  this  study. 

Self-Efficacy.  Self-efficacy  refers  to  the  belief  that  one  has  in  his  ability 
to  confidently  perform  a  specific  task  (Bandura,  1991).  As  reported  by  Salas  and 
Cannon-Bowers  (2001),  several  studies  have  identified  self-efficacy  as  a  strong  indicator 
of  performance  (Mathieu  et  ah,  1993;  Mathieu  et  ah,  1992;  Stajkovic  &  Luthans,  1998). 
Additionally,  Tannenbaum  and  Yukl  (1992)  and  Mathieu  et  al.  (1992)  have  discussed  the 
importance  of  self-efficacy  as  an  antecedent  to  and  outcome  of  training.  As  a  measure  of 
training  effectiveness,  both  task-related  self-efficacy  and  learning  self-efficacy  are 
important. 

Task-related  (i.e.,  the  specific  tasks  in  which  trainees  are  instructed)  self-efficacy 
measurement  has  been  supported  in  several  research  studies.  Ford  et  al.’s  (1992) 
research  supported  the  concept  of  self-efficacy  and  its  relationship  with  motivation  to 
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transfer  concluding  that  higher  self-efficacy  resulted  in  individuals  performing  more  of 
the  tasks  they  learned  and  at  a  more  complex  and  difficult  level.  Stajkovic  and  Luthans 
(1998)  conducted  research  that  identified  a  significant  positive  correlation  between  self- 
efficacy  and  work-related  performance.  Mager  (1992)  emphasized  the  importance  of 
strengthening  self-efficacy  for  successful  job  performance.  He  concluded  that  skills 
unaccompanied  by  positive  self-efficacy  would  lead  to  deficient  or  absent  performance. 
Frayne  and  Latham  (1987)  conducted  a  study  of  state  government  employees  that 
revealed  higher  performance  (i.e.,  higher  job  attendance  -  the  outcome  expected  in  the 
study)  was  a  result  of  higher  perceived  self-efficacy.  Robertson  and  Sadri  (1993) 
performed  a  study  of  task-related  self-efficacy  that  focused  on  managerial  skills  and  work 
performance.  They  predicted  a  positive  correlation  between  task-related  self-efficacy  and 
work  performance  based  on  previous  studies  identified  in  their  literature  review.  The 
results  of  their  study  confirmed  their  prediction.  Robertson  and  Sadri  further  discussed 
the  importance  of  measuring  self-efficacy  immediately  after  training  as  an  early 
indication  of  job  performance  as  well  as  a  valuable  assessment  of  training  effectiveness. 

Numerous  studies  have  also  supported  learning  or  academic  self-efficacy.  Salas 
and  Cannon-Bowers  (2001)  provided  support  for  learning  self-efficacy  predicting 
performance,  mediating  other  variables,  and  enhancing  learning  outcomes.  Tannenbaum 
and  Yukl  (1992)  acknowledged  that  trainee  self-efficacy  was  another  important  construct 
and  that  individuals  with  high  self-efficacy  tended  to  outperform  individuals  with  low 
self-efficacy.  Further,  self-efficacy  can  be  a  predictor  of  training  success  or  even  a 
desirable  outcome  of  training  (Tannenbaum  &  Yukl,  1992).  Mathieu  et  al.  (1993) 
conducted  research  to  identify  antecedents  that  affect  self-efficacy  development  during 
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training.  Results  indicated  the  self-efficacy  had  significant  positive  influences  on  training 
reactions  and  subsequent  performance.  In  Warr  and  Bunce’s  (1995)  research,  learning 
self-efficacy  was  shown  to  be  significantly  related  to  learning  score.  In  their  research, 
Warr  and  Bunce  identified  the  importance  of  learning  self-efficacy  and  its  expectation  to 
be  positively  related  to  learning  performance.  Kraiger  et  al.  (1993)  proposed  that 
changes  in  trainees’  self-efficacy  may  be  a  useful  indicator  of  learning.  Additionally, 
enhanced  self-efficacy  may  be  a  formal  training  objective,  moderating  the  relationship 
between  learning  and  performance.  Further,  post-training  self-efficacy  beliefs  may  be 
useful  indicators  of  transfer  and  changes  in  self-efficacy  may  infer  evidence  of 
development  during  training  (Kraiger  et  ah,  1993).  Colquitt  et  al.’s  (2000)  meta-analysis 
further  supported  the  idea  revealed  in  previous  research  that  a  positive  relationship 
between  self-efficacy  and  training  outcomes  exists. 

Research  on  self-efficacy  in  training  is  prevalent  and  measurement  of  the 
construct  is  supported  on  a  widespread  basis.  Two  facets  of  self-efficacy,  learning  and 
task-related,  were  measured  in  this  study  because  both  have  been  shown  to  predict 
performance  outcome. 

Training  Motivation. 

Another  significant  construct  in  training  evaluation  is  training  motivation  or 
motivation  to  learn  by  the  trainee.  Training  motivation  has  effects  before,  during,  and 
after  training  (Salas  &  Cannon-Bowers,  2001).  Baldwin,  Magjuka,  and  Loher  (1991) 
performed  research  that  revealed  how  choice  of  training  may  influence  trainee 
motivation.  Baldwin  et  al.  found  that  trainees  who  neither  received  nor  had  a  choice  in 
training  were  less  motivated  and  thus  performed  at  a  lower  level  than  those  trainees  who 
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received  their  choice  of  training.  While  the  “choice  of  training”  antecedent  was  not 
included  in  this  research,  Baldwin  et  al.’s  study  supports  the  need  to  study  which 
antecedents  influence  trainee  motivation  and  the  relationship  training  motivation  has  to 
performance.  Noe’s  (1986)  review  of  training  literature  led  to  the  conclusion  that 
learning  motivation  is  a  direct  antecedent  to  learning.  Noe  and  Schmitt  (1986)  also 
mentioned  the  relationship  between  motivation  to  learn  and  learning  identified  in 
previous  research  and  included  the  construct  in  their  exploratory  model  of  motivational 
influences  on  training  effectiveness.  The  purpose  of  their  research  was  to  investigate 
relationships  between  training  effectiveness  and  training  attitudes.  Further  research 
indicates  that  motivated  trainees  take  a  more  active  role  in  training  thus  gaining  more 
experience  than  individual  with  low  motivation  levels  (Tracey  &  Tews,  1995).  It  is  also 
acknowledged  in  the  literature  that  there  is  a  wide  acceptance  of  trainees  learning  and 
transferring  what  they’ve  learned  when  motivation  is  high  (Tannenbaum  &  Yukl,  1992). 
Warr  and  Bunce  (1995)  identified  an  individual’s  motivation  to  learn  as  an  important 
determinant  of  training  outcomes.  Colquitt  et  al.  (2000)  identified  training  motivation  as 
a  function  of  variables  such  as  organizational  commitment  suggesting  that  further 
research  of  training  motivation  is  needed.  Because  of  the  diverse  effects  training 
motivation  may  have  before,  during,  and  after  training,  the  training  motivation  construct 
was  included  in  this  study. 

Trainee  Demographics. 

According  to  Colquitt  et  al.  (2000),  demographics  refers  to  the  ascribed  or 
achieved  characteristics  of  individuals.  They  have  identified  that  little  theory  exists 
linking  demographics  to  training  outcome.  Further,  they  state  that  demographics  are 
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most  often  used  as  statistieal  eontrol  variables.  To  further  the  limited  researeh  in  the  area 


of  demographics  and  their  link  with  training  outcomes,  this  study  solicited  age,  gender, 
rank,  experience,  assigned  MAJCOM,  and  time  in  service  from  the  research  participants 
and  suggested  relationships  between  demographics  and  pre-training  trainee 
characteristics  that  have  been  identified  in  the  literature  below. 

Age.  Empirical  studies  of  age  and  training  outcomes  have  been  more 
consistent  than  other  demographic  information  collected  (Colquitt  et  ah,  2000).  Many 
studies  have  revealed  a  negative  relationship  between  age  and  learning.  Older  trainees 
demonstrated  lower  motivation,  learning,  and  post-training  self-efficacy.  Warr  and 
Bunce  (1995)  also  studied  the  relationship  that  age  has  with  learning.  They  suggested 
that  an  age  gradient  to  learning  performance  existed.  Noe,  Wilk,  Mullen,  and  Wanek 
(1997)  discussed  the  relationship  between  age  and  motivation  to  participate  in  employee 
development  programs.  Research  indicated  that  younger  workers  were  more  willing  to 
engage  in  employee  development.  This  study  suggests  that  older  employees  may  have 
less  training  motivation  and  self-efficacy  during  training. 

Gender.  Studies  of  gender  and  its  effects  on  learning  have  been 
ambiguous  (Colquitt  et  ah,  2000).  Colquitt  et  al.  state  that  failure  to  find  consistent 
effects  for  gender  on  learning  is  not  surprising  given  the  lack  of  theory  for  such  effects. 
Noe  et  al.  (1997)  noted  observations  where  gender  had  a  significant  impact  in  certain 
situations.  For  example,  women’s  career  paths  are  much  more  complex  and  do  not 
follow  traditional  models.  Women  are  not  offered  the  same  opportunities  to  develop 
within  organizations.  Women  also  reported  receiving  less  support  than  men  from  others 
in  the  work  environment  as  being  an  obstacle  in  participating  in  development  activities. 
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Taken  in  a  training  context,  when  given  the  opportunity  to  participate  in  employee 
development,  this  study  asks  if  there  are  correlations  between  gender  and  training 
outcomes. 

Prior  Job  Knowledge.  Smith- Jentsch,  Jentsch,  Payne,  and  Salas  (1996) 
conducted  a  study  to  examine  effects  of  pre-training  knowledge  on  performance.  Results 
indicated  a  linear  relationship  between  prior  knowledge  and  performance.  The  study 
suggested  that  participants  with  pre -training  knowledge  had  more  motivation  to  learn 
than  participants  without  prior  knowledge  (Smith- Jentsch  et  ah,  1996).  Warr  and  Bunce 
(1995)  identified  that  prior  job  experience  may  be  a  factor  in  influencing  learning.  Their 
research  showed  a  positive  relationship  between  previous  experience  and  work  outcomes. 
However,  it  was  undetermined  whether  lengthy  experience  in  a  particular  job  would  have 
learning  benefits.  Warr  and  Bunce  identified  this  area  as  having  a  need  for  further 
research.  Ree,  Carretta,  and  Teachout  (1995)  performed  research  that  included  studying 
the  relationship  of  prior  job  knowledge  and  learning  during  training.  Results  of  the  study 
indicated  that  prior  job  knowledge  had  little  influence  on  subsequent  job  knowledge,  but 
direct  influence  on  early  work  which  in  turn  influenced  performance.  This  indirect 
impact  on  performance  through  prior  job  knowledge  provides  reason  for  studying 
correlations  between  prior  job  knowledge  and  other  trainee  characteristics.  Trainees  with 
prior  job  knowledge  should  have  better  performance  in  training  (Ree  et  ah,  1995). 
Colquitt  et  al.  (2000)  and  Salas  and  Cannon-Bowers  (2001)  support  the  idea  identified  in 
their  research  reviews  that  trainees’  prior  job  knowledge  has  significant  importance 
during  the  training  intervention. 
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Environment/Climate.  Support  for  the  influence  of  learning  outcomes  and 
of  the  transfer  climate  on  using  knowledge,  skills,  and  attitudes  back  on  the  job  appears 
throughout  the  training  literature  (Salas  &  Cannon-Bowers,  2001).  Factors  in  the  work 
environment  may  enhance  or  inhibit  transfer  of  training  (Ford  et  ah,  1992).  Tracey, 
Tannenbaum,  and  Kavanagh’s  (1995)  research  supports  studying  how  the  work 
environment  influences  trainee  perceptions  and  behavior.  They  suggested  that  climate 
may  have  a  direct  effect  on  self-efficacy  and  motivation  to  learn.  Further,  trainees  in  a 
less  supportive  work  environment  will  be  less  likely  to  acquire  new  knowledge  gained 
from  any  means,  formal  training  or  otherwise.  Colquitt  et  al.’s  (2000)  review  of  training 
research  found  that  climate  may  predict  the  extent  to  which  trainee’s  transfer  knowledge, 
skills,  and  attitudes  to  the  job.  Colquitt  et  al’s  review  revealed  that  a  positive  relationship 
existed  in  several  studies  between  the  organization’s  climate  and  the  transfer  of  learning 
to  the  job.  However,  Colquitt  et  al.  noted  that  examination  of  situational  characteristics  is 
still  rare.  Research  is  needed  to  identify  which  facets  of  climate,  culture,  and  context 
have  the  most  positive  relationships  with  training  outcomes.  This  study  classified  the 
trainee’s  command  assignment  as  the  environment  in  which  the  trainee  would  work. 

Time  in  Service.  Warr  and  Bunce  (1995)  studied  this  individual  factor  to 
determine  if  there  was  an  association  between  job  tenure  and  learning.  In  their  research, 
they  made  no  prediction  of  the  effect  of  job  tenure  on  learning.  Results  of  their  research 
indicated  that  job  tenure  may  have  a  negative  correlation  with  other  constructs  identified 
to  study  training  effectiveness.  This  study  seeks  to  examine  the  relationship  between 
time  in  service  and  trainee  characteristics. 


21 


Organization  Characteristics 

Organization  Support. 

Due  to  differences  in  organizations  (e.g.,  the  organization’s  mission),  the  level  of 
support  received  from  the  organization  may  have  an  effect  on  the  trainees’  opportunity  to 
perform  and  subsequent  transfer  of  skills  learned  to  the  work  environment  (Ford  et  ah, 
1992).  Additionally,  trainees  may  feel  more  comfortable  performing  learned  tasks  in  a 
highly  supportive  environment.  Organization  support  is  also  believed  to  promote 
participation  in  employee  development  activities  (Noe  et  ah,  1997).  Tracey  et  al.  (1995) 
revealed  a  consistent  theme  in  their  literature  review  that  work  environment  may  impact 
one’s  preparation  for  training  and  subsequently  the  transfer  of  training  back  to  the  work 
environment.  Noe  and  Schmitt  (1986)  substantiated  the  idea  that  a  supportive  work 
environment  would  increase  transfer  of  skills  to  the  work  environment.  Finally,  support 
for  further  research  in  the  area  of  organization  support  was  discussed  by  Tannenbaum 
and  Yukl  (1992)  because  of  the  possibility  that  the  organization  climate  may  be  an 
important  determinant  of  training  effectiveness. 

Organizational  Constraints. 

One  underlying  theme  throughout  the  training  literature  is  the  idea  that 
organizational  constraints  may  impact  performance.  Constraints  refer  to  any  inhibitor 
perceived  by  a  person  and  may  differ  from  one  person  to  another  (Mathieu  et  ah,  1993). 
Organizational  constraints  are  believed  to  inhibit  participation  in  employee  development 
activities  (Noe  et  ah,  1997).  Noe’s  (1986)  analysis  of  training  literature  uncovered  the 
concept  that  trainee’s  perceptions  of  the  favorability  of  the  work  environment  influence 
motivation  to  learn  and  transfer  of  skills.  Other  research  supports  organizational 
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constraint  measurement  in  determining  the  impact  on  performance.  Mathieu  et  al.  (1992) 
researehed  a  model  that  included  a  hypothesized  relationship  between  performanee  and 
organizational  constraints.  Mathieu  et  al.  eoneluded  that  if  trainees  believed  that  learning 
new  skills  would  not  add  value  to  their  job  performance  due  to  organizational  constraints, 
trainees  would  have  less  motivation  to  perform  well  in  training.  O’Connor  et  al.  (1984) 
condueted  a  field  investigation  of  the  impaet  of  situational  constraints  on  individual 
performance.  They  found  signifieant  association  between  inhibiting  situational 
constraints  and  lower  performance.  Peters,  O’Connor,  Eulberg  and  Watson’s  (1988) 
researeh  proposed  that  situational  eonstraints  eould  limit  individual  work  performance. 
However,  in  their  research,  constraints  were  not  shown  to  relate  to  performance  and 
therefore,  left  a  requisite  to  further  the  understanding  of  the  determinants  of  performanee. 
Hypotheses  of  the  importanee  of  measuring  work-related  situational  eonstraints  that 
direetly  or  indirectly  attribute  to  explaining  the  varianee  in  performanee,  ineluding  those 
whieh  affeet  training  and  development  programs  are  also  prevalent  (Peters  &  O'Connor, 
1980;  Peters,  O'Connor,  &  Eulberg,  1985).  Two  hypotheses  relevant  to  this  researeh 
inelude:  a)  situational  constraints  having  a  negative  impaet  on  performanee  and  b) 
individual  differences  in  abilities  and  motivation  interacting  with  situational  constraints 
in  the  predietion  of  performanee.  Einally,  according  to  Tannenbaum  and  Yukl  (1992), 
limitations  in  the  work  environment  can  also  influence  the  events  that  occur  after  a 
trainee  returns  to  the  job.  This  study  is  interested  in  relationships  that  may  exist  between 
perceived  organizational  constraints  and  training  outcomes. 
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Opportunity  to  Perform. 

One  issue  that  can  affect  the  transfer  of  skills  to  the  job  is  the  opportunity  the 
trainee  is  given  to  perform  the  task  which  relates  to  the  organization  climate  and  the 
trainee’s  self-efficacy  (Ford  et  ah,  1992).  In  a  study  of  trainee  attitudes  and  their  effect  on 
training  effectiveness,  Noe  and  Schmitt  (1986)  included  the  trainee’s  perception  of 
opportunity  to  perform  in  their  exploratory  model.  They  identified  that  the  trainee’s 
belief  regarding  opportunity  to  perform  skills  or  to  use  knowledge  learned  in  training 
programs  are  of  particular  importance  in  evaluating  training  effectiveness.  The  untested 
assumption  that  all  trainees  will  have  the  same  opportunity  to  perform  tasks  learned  has 
not  been  extensively  measured.  In  the  context  of  this  study,  opportunity  to  perform  is 
defined  as  the  extent  to  which  the  trainee  feels  he  will  have  the  opportunity  to  use  the 
knowledge  learned  in  BCOT  on  the  job  (Ford  et  ah,  1992).  This  study  measured  the 
trainees  perception  of  whether  they  will  have  the  opportunity  to  perform  each  of  the 
eleven  course  goals  learned.  Given  the  opportunity  to  perform  the  task,  will  the 
motivation  to  transfer  the  learned  tasks  correlate  with  the  trainees’  perceptions? 

Training  Outcomes 

Training  Performance. 

Numerous  methods  are  used  to  measure  learning  and  immediate  post-training 
knowledge  (Alliger  &  Tannenbaum,  1996).  Kirkpatrick  (1978)  acknowledged  that  it  is 
appropriate  to  measure  knowledge  or  attitude  change  with  a  paper-and-pencil  test.  He 
suggested  the  appropriateness  of  tailoring  the  test  to  cover  the  specific  knowledge  being 
taught.  Any  program  where  skills  of  some  kind  are  being  taught  should  use  a  systematic 
classroom  evaluation  to  measure  learning  (Kirkpatrick,  1979).  An  analysis  of  techniques 
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to  measure  learning  identified  that  measurement  most  often  took  place  by  a  pre-post  test 
measurement  design  as  well  as  by  paper-and-pencil  tests  to  determine  knowledge  and 
understanding  of  concepts  taught  (Catalanello  &  Kirkpatrick,  1968).  Alliger  et  al.’s 
(1997)  meta-analysis  of  34  studies  also  revealed  that,  by  far,  the  most  common 
measurement  of  immediate  post-training  knowledge  was  traditional  tests  such  as  multiple 
choice  paper-and-pencil  tests.  Another  consideration  for  training  performance  is  when  to 
measure.  Kraiger  et  al.  (1993)  identified  that  measuring  trainee’s  retention  of  declarative 
knowledge  is  most  appropriate  in  the  initial  stages  of  training.  In  the  BCOT  course, 
grades,  based  on  a  lOO-point  scale,  are  given  after  each  block  of  instruction.  Some  block 
grades  are  from  paper-and-pencil  tests;  while  others  are  from  projects  performed  in  the 
block.  This  study  used  the  final  course  score  (an  average  of  the  grades  for  each  of  the 
eleven  blocks  of  training)  for  each  trainee  as  a  measure  of  training  performance. 

Training  Reaction. 

Research  has  shown  a  relationship  between  training  participants’  reactions  and 
their  behavioral  change  following  training  course  completion  (Faerman  &  Ban,  1993). 
Kirkpatrick  (1978)  iterated  that  reactions  measurement  should  be  done  in  a  way  so  that 
the  results  may  be  tabulated  and  quantified.  Warr  and  Bunce  (1995)  supported 
measurement  of  three  kinds  of  reaction:  enjoyment,  usefulness,  and  difficulty.  No 
relationship  between  perceived  enjoyment  reaction  and  learning  was  expected  nor  found. 
Perceived  usefulness,  however,  was  more  likely  to  be  associated  with  changes  in  work 
behavior  because  trainees  who  saw  training  as  being  more  relevant  to  their  work  would 
likely  want  to  transfer  the  learning  to  their  work  environment.  Warr  and  Bunce’s 
research  results  showed  a  positive  correlation  between  training  usefulness  and 
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motivation.  Perceived  difficulty  was  expected  to  predict  immediate  learning.  Research 
results  showed  a  negative  correlation  between  the  two;  however,  this  may  have  been 
because  of  the  open  learning  forum  that  was  used  in  the  study.  All  three  levels  of 
reaction  were  seen  as  important  for  measuring  training  outcomes.  Warr  and  Bunce 
further  stated  that  predictors  of  reaction  to  training  are  dependent  on  which  form  of 
reaction  is  measured.  Alliger  and  Tannenbaum  (1996)  identified  the  difference  between 
affective  and  utility  reaction.  Their  research  supported  correlations  between  reaction  and 
job  application.  Their  research  results  supported  utility  over  affective  reaction  and  found 
that  utility  reaction  correlated  more  highly  with  on-the-job  performance  than  with 
affective  reaction.  They  concluded  that  measuring  utility  reaction  would  provide  a  better 
estimate  of  transfer.  In  a  review  of  training  literature,  Alliger  et  al.  (1997)  performed  a 
meta-analysis  of  34  studies  that  yielded  115  correlations  between  training  criteria.  Their 
findings  also  supported  that  utility-type  reaction  was  more  strongly  related  to  learning  or 
transfer  than  affective  reactions. 

This  study  measured  all  three  types  of  reactions  from  training  and  analyzed  each 
reaction  against  other  training  outcomes  as  well  as  combining  all  three  reactions  into  a 
single  construct  for  comparison  to  either  support  or  not  support  the  measurement  of  three 
separate  reactions  that  has  been  identified  in  previous  research. 

Motivation  to  Transfer. 

Training  literature  not  only  supports  motivation  to  transfer  as  an  important 
outcome  of  training,  but  also  identifies  that  there  are  numerous  training  constructs  that 
may  affect  motivation  to  transfer  (Salas  &  Cannon-Bowers,  2001).  Noe  (1986)  defines 
motivation  to  transfer  as  the  trainees’  desire  to  use  the  knowledge  and  skills  mastered  in 


26 


the  training  program.  Noe  and  Sehmitt  (1986)  identified  motivation  to  transfer  as  an 
outcome  of  training  and  a  measure  of  training  effectiveness.  Motivation  to  transfer  has 
been  shown  as  an  indication  of  success  (Alliger  &  Tannenbaum,  1996).  Ford  et  al. 

(1992)  posited  that  the  transfer  of  trained  tasks  to  the  job  will  be  successful  if  given  the 
opportunity  to  perform  those  tasks.  Tracey  et  al.’s  (1995)  research  showed  that  the  work 
environment  was  an  important  contributor  to  the  application  of  newly  acquired  behavior 
and  skills.  Kraiger  et  al.  (1993)  identified  previous  training  effectiveness  models  that 
treated  learning  as  a  unidimensional  construct  where  different  learning  outcomes  were 
not  defined,  but  learning  was  recognized  an  important  pre-cursor  to  transfer.  Kozlowski 
and  Salas  (1997)  identified  a  high  consensus  that  training  is  of  little  value  if  the 
knowledge,  skills,  behaviors,  and  attitudes  acquired  by  the  trainee  are  not  transferred  to 
the  job  setting.  While  acknowledging  the  importance  of  motivation  to  transfer  has  been 
consistent,  too  little  attention  has  been  given  to  assess  transfer  of  learned  knowledge, 
skills,  and  attitudes  to  the  job  (Faerman  &  Ban,  1993).  This  study  seeks  to  examine 
motivation  to  transfer  as  a  desired  training  outcome  and  a  possible  indicator  for  future  job 
performance. 

This  section  has  identified  the  pertinent  training  constructs  identified  in  the 
training  literature  that  may  be  relevant  in  measuring  training  effectiveness.  The  next 
section  will  discuss  previous  research  of  training  effectiveness  models. 

Prior  Training  Effectiveness  Models 

According  to  Alliger  and  Janak  (1989),  Kirkpatrick’s  model  has  met  an 
organizational  need  for  evaluating  training.  Admittedly,  Kirkpatrick’s  model  has  been 
good  for  evaluation  in  the  past  (Watkins  et  al.,  1998).  Support  for  using  Kirkpatrick’s 
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simple,  yet  systematic  evaluation  has  abounded;  yet  there  remains  a  need  to  expand 
Kirkpatrick’s  model  to  include  other  pertinent  constructs  that  may  broaden  the 
understanding  of  training  effectiveness.  For  example,  Alliger  et  al.  (1997)  suggested  that 
there  is  a  broader  understanding  of  training  evaluation  needed  by  using  training  reaction 
to  measure  learning  and  transfer.  Salas  and  Cannon-Bowers  (2001)  discussed  the 
Kirkpatrick  model  weaknesses  and  suggested  research  in  the  area  of  training  antecedents 
and  their  impact  on  training  effectiveness.  Warr  and  Bunce  (1995)  identified  a  need  to 
study  additional  associations  between  immediate  learning  and  later  outcomes  from 
learning.  Additionally,  Miller  (1990)  suggested  that  training  and  development  of 
personnel  can  be  used  as  a  tool  for  furthering  organization  development  supporting  the 
idea  of  using  a  systems  approach  to  look  at  training.  This  section  will  present  recent 
training  evaluation  and  effectiveness  research  that  was  used  as  the  foundation  for  this 
study. 

Noe  and  Schmitt  (1986)  conducted  a  study  to  test  an  exploratory  model  of  the 
influence  of  trainee  attitudes  concerning  their  jobs  and  trainee  perception  of  the 
organization  climate  on  training  outcomes.  The  study  provided  limited  support  for 
Kirkpatrick’s  model.  In  addition,  the  positive  findings  of  their  study  supported 
expanding  training  effectiveness  measurement  to  include  trainee  and  organization 
characteristics  and  provided  future  direction  for  additional  research  of  trainee  attitudes 
which  may  attenuate  or  enhance  training  effectiveness.  They  identified  the  influence  of 
employee  reactions  on  motivation  to  learn  during  training  and  trainee  perceptions  of 
organization  support  as  two  important  areas  for  future  research. 
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Research  conducted  by  Tannenbaum  et  al.  (1991)  studied  training  characteristics 
and  the  effect  training  fulfillment  had  on  those  characteristics.  The  research  hypotheses 
stated  that  training  fulfillment,  trainee  reactions,  and  training  performance  would  be 
related  to  the  development  of  post-training  attitudes.  They  identified  commitment,  self- 
efficacy,  and  motivation  as  antecedents  and  outcomes  of  training.  The  results  supported 
each  of  the  hypotheses.  Training  fulfillment  positively  related  to  post- training  attitudes 
and  pre-training  attitudes  related  to  the  development  of  post-training  attitudes.  The 
results  of  this  research  furthered  the  belief  that  commitment,  self-efficacy,  and  motivation 
are  important  antecedents  and  outcomes  of  training  and  implied  that  training  fulfillment 
enhances  each.  In  addition,  Tannenbaum  et  al.  mentioned  that  developing  these  three 
trainee  characteristics  may  result  in  higher  performance  and  motivation  to  transfer, 
concepts  for  future  research  to  expand  the  Tannenbaum  et  al. study.  Tannenbaum  et  al.’s 
study  supplied  further  support  to  measure  commitment,  self-efficacy,  and  motivation  of 
the  trainee  and  to  identify  how  these  trainee  characteristics  correlate  with  other  training 
effectiveness  constructs. 

Theoretical  support  for  expanding  the  factors  of  training  effectiveness  to  include 
trainee  characteristics  is  provided  by  Tannenbaum  and  Yukl  (1992).  Implications  for 
improving  training  effectiveness  included  measuring  trainee  characteristics  such  as 
motivation,  attitudes,  abilities,  skills,  and  aptitude  treatment  interactions.  They  stated  that 
self-efficacy  and  motivation  are  central  constructs  in  understanding  training 
effectiveness.  These  characteristics  can  be  influenced  and  influential  before,  during,  and 
after  the  training  process.  Tannenbaum  and  Yukl  emphasized  that  because  of  the 
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potential  to  help  in  explaining  why  training  works,  these  constructs  should  receive 
particular  attention  during  research. 

Mathieu  et  al.  (1992)  performed  a  study  that  hypothesized  relationships  between 
individual  and  situational  characteristics  and  training  motivation  and  training 
effectiveness.  Their  findings  supported  relationships  between  learning  and  performance; 
and  training  motivation  and  reactions.  They  also  identified  antecedents  of  training 
motivation.  Mathieu  et  al.  revised  their  model  to  show  the  complex  role  those  reactions 
to  training  play  in  measuring  training  effectiveness.  Further,  they  suggested  replication 
of  the  study  in  other  settings  and  other  training  programs. 

Cannon-Bowers  et  al.  (1995)  performed  a  study  in  a  military  setting  in  which  the 
results  indicated  that  several  non-technical  trainee-related  factors  had  significant  impact 
on  training  outcomes.  They  acknowledged  that  training  effectiveness  is  a  complex 
phenomena  and  that  training  variables  are  a  critical  part  of  the  effectiveness  equation. 
Their  research  studied  pieces  of  a  systems-oriented  comprehensive  model  in  which 
organization  and  trainee  characteristics  were  crucial  inputs  for  measuring  training  results. 
They  selectively  chose  the  more  promising  variables  identified  in  the  literature,  such  as 
motivation,  self-efficacy,  and  expectation  variables,  to  assess  their  impact  on  training 
effectiveness. 

Mathieu  and  Martineau  (1997)  proposed  a  conceptual  model  (Figure  1)  to  posit 
implications  of  individual  and  situational  influences  on  training  motivation.  Mathieu  and 
Martineau  emphasized  expanding  Kirkpatrick’s  framework  beyond  the  immediate 
training  program  to  provide  a  more  complete  understanding  of  training  effectiveness. 
While  Mathieu  and  Martineau’ s  discussion  focused  on  training  motivation,  they  also 
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pointed  out  that  self-efficacy  is  an  important  predictor  of  training  outcomes,  specifically 
training  transfer.  Additionally,  Mathieu  and  Martineau  identified  individual  influences 
such  as  demographics;  knowledge,  skills,  abilities,  and  experiences;  personality  and 
needs;  and  work-related  attitudes  that  may  affect  training  motivation,  in  turn,  influencing 
training  effectiveness.  Situational  influences  such  as  constraints,  social-psychological 
influences,  and  maintenance  systems  were  also  predicted  to  have  influences  on 
motivation  as  well  as  training  effectiveness.  While  unable  to  test  their  conceptual  model, 
Mathieu  and  Martineau  presented  implications  for  research  that  focus  on  the  complete 
context  of  training  when  evaluating  effectiveness  of  training  programs.  They  also  stress 
the  use  of  a  systems-oriented  approach  in  evaluation  as  well  as  training  design. 

Therefore,  according  to  Mathieu  and  Martineau,  the  training  system  should  be  viewed  in 
the  context  of  ongoing  organizational  processes,  and  the  effectiveness  of  training  depends 
on  the  program  as  well  as  relevant  individual  and  situational  factors. 


Figure  1.  Mathieu  and  Martineau  (1997)  Conceptual  Model 
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Research  Model 


Based  on  previous  researeh  and  the  eoneeptual  model  (Figure  1)  proposed  by 
Mathieu  and  Martineau  (1997),  ten  construets  (i.e.,  affective  organization  commitment, 
learning  and  task-related  self-efficacy,  training  motivation,  organization  support, 
organization  constraints,  opportunity  to  perform,  training  performance,  training  reaction, 
and  motivation  to  transfer),  as  well  as  certain  trainee  demographics,  were  chosen  to 
examine  the  impact  of  trainee  and  organization  characteristics  on  the  training  outcomes 
of  the  BCOT  course.  The  proposed  research  model  for  this  study  is  depicted  in  Figure  2. 
The  proposed  research  model  identifies  overarching  relationships  between  trainee 
characteristics,  organization  characteristics,  and  training  outcomes.  The  following 
section  presents  the  hypotheses  posited  to  answer  the  research  questions  identified  in 
Chapter  1. 


Figure  2.  Proposed  Research  Model 
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Based  on  research  supporting  the  ten  training  constructs  selected  for  study  and  the 
proposed  research  model,  the  following  hypotheses  are  presented: 

Hypothesis  1  -  Hypotheses  to  support  relationships  between  trainee 
characteristics  and  training  outcomes. 

HI  a:  There  is  a  positive  relationship  between  organization  commitment  and 
training  performance. 

Hlb:  There  is  a  positive  relationship  between  task-related  self-efficacy  and 
training  performance. 

Hlc:  There  is  a  positive  relationship  between  learning  self-efficacy  and  training 
performance. 

Hid:  There  is  a  positive  relationship  between  motivation  to  learn  and  training 
performance. 

Hie:  There  is  a  positive  relationship  between  organization  commitment  and 
training  reactions. 

Hlf:  There  is  a  positive  relationship  between  task-related  self-efficacy  and 
training  reactions. 

Hlg:  There  is  a  positive  relationship  between  learning  self-efficacy  and  training 
reactions. 

Hlh:  There  is  a  positive  relationship  between  motivation  to  learn  and  training 
reactions. 

Hli:  There  is  a  positive  relationship  between  organization  commitment  and 
motivation  to  transfer. 
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Hlj:  There  is  a  positive  relationship  between  task-related  self-efficacy  and 
motivation  to  transfer. 

Hlk:  There  is  a  positive  relationship  between  learning  self-efficacy  and 
motivation  to  transfer. 

Hll:  There  is  a  positive  relationship  between  motivation  to  learn  and  motivation 
to  transfer. 

Figure  3  pictorially  represents  the  proposed  relationships  for  hypothesis  1 . 


Pre-Training  Trainee 


Hypothesis  2  -  Hypotheses  to  support  relationships  between  organization 
characteristics  and  training  outcomes. 

H2a:  There  is  a  positive  relationship  between  organization  support  and  training 
performance. 
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H2b:  There  is  a  negative  relationship  between  organization  constraints  and 
training  performance. 

H2c:  There  is  a  positive  relationship  between  opportunity  to  perform  and 
training  performance. 

H2d:  There  is  a  positive  relationship  between  organization  support  and  training 
reactions. 

H2e:  There  is  a  negative  relationship  between  organization  constraints  and 
training  reactions. 

H2f:  There  is  a  positive  relationship  between  opportunity  to  perform  and 
training  reactions. 

H2g:  There  is  a  positive  relationship  between  organization  support  and 
motivation  to  transfer. 

H2h:  There  is  a  negative  relationship  between  organization  constraints  and 
motivation  to  transfer. 

H2i:  There  is  a  positive  relationship  between  opportunity  to  perform  and 
motivation  to  transfer. 

Figure  4  pictorially  represents  the  proposed  relationships  for  hypothesis  2. 
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Training  Outcomes 

Training  performance 

Training  reactions 

Motivation  to  transfer 

Figure  4.  Proposed  Hypothesis  2  Relationships 

Hypothesis  3  -  Hypotheses  to  support  relationships  between  trainee 
demographics  and  trainee  characteristics. 

H3a:  There  is  a  positive  relationship  between  age  and  organization  commitment. 
H3b:  There  is  a  relationship  between  gender  and  organization  commitment. 
H3c:  There  is  a  relationship  between  MAJCOM  assignment  and  organization 
commitment. 

H3d:  There  is  a  positive  relationship  between  time  in  service  and  organization 
commitment. 

H3e:  There  is  a  positive  relationship  between  previous  communications 
experience  and  organization  commitment. 

H3f:  There  is  a  positive  relationship  between  age  and  task-related  self-efficacy. 
H3g:  There  is  a  relationship  between  gender  and  task-related  self-efficacy. 

H3h:  There  is  a  relationship  between  MAJCOM  assignment  and  task-related 
self-efficacy. 


Organization 

Characteristics 
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H3i:  There  is  a  positive  relationship  between  time  in  service  and  task-related 
self-efficacy. 

H3j:  There  is  a  positive  relationship  between  previous  communications 
experience  and  task-related  self-ejficacy. 

H3k:  There  is  a  positive  relationship  between  age  and  learning  self-ejficacy. 
H31:  There  is  a  relationship  between  gender  and  learning  self-efficacy. 

H3m:  There  is  a  relationship  between  MAJCOM  assignment  and  learning  self- 
efficacy. 

H3n:  There  is  a  positive  relationship  between  time  in  service  and  learning  self- 
ejficacy. 

H3o:  There  is  a  positive  relationship  between  previous  communications 
experience  and  learning  self-efficacy. 

H3p:  There  is  a  negative  relationship  between  age  and  motivation  to  learn. 
H3q:  There  is  a  relationship  between  gender  and  motivation  to  learn. 

H3r:  There  is  a  relationship  between  MAJCOM  assignment  and  motivation  to 
learn. 

H3s:  There  is  a  positive  relationship  between  time  in  service  and  motivation  to 
learn. 

H3t:  There  is  a  positive  relationship  between  previous  communications 
experience  and  motivation  to  learn. 

Figure  5  pictorially  represents  the  proposed  relationships  for  hypothesis  3. 
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Pre-Training  Trainee 
Characteristics 


Organization  commitment 


Task-reiated  seif-efficacy 


Learning  seif- efficacy 


Motivation  to  iearn 


Figure  5.  Proposed  Hypothesis  3  Relationships 


Hypothesis  4  -  Hypotheses  to  support  relationships  between  pre-training  trainee 
characteristics  and  post-training  characteristics. 

H4a:  There  will  be  an  increase  in  organization  commitment  from  pre-training  to 
post-training. 

H4b:  There  will  be  an  increase  in  task-related  self-efficacy  from  pre-training  to 
post-training. 

Hypothesis  5  -  Hypotheses  supporting  relationships  between  training  outcomes 
and  post-training  characteristics. 

H5a:  There  is  a  positive  relationship  between  training  performance  and  post¬ 
training  organization  commitment. 
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H5b:  There  is  a  positive  relationship  between  training  reactions  and  post¬ 
training  organization  commitment. 

H5c:  There  is  a  positive  relationship  between  motivation  to  transfer  and  post¬ 
training  organization  commitment. 

H5d:  There  is  a  positive  relationship  between  training  performance  and  post¬ 
training  task-related  self-efficacy. 

H5e:  There  is  a  positive  relationship  between  training  reactions  and  post¬ 
training  task-related  self-efficacy. 

H5f  There  is  a  positive  relationship  between  motivation  to  transfer  and  post¬ 
training  task-related  self-efficacy. 

Figure  6  pictorially  represents  the  proposed  relationships  for  hypotheses  5. 


Training  Outcomes 


Post-Training  Trainee 
Characteristics 


Training  performance 


Training  reactions 


Motivation  to  transfer 


+ 

+ 

+ 

+ 

+ 

+ 


Organization  commitment 


Task-related  self-eflTicacy 


Figure  6.  Proposed  Hypothesis  5  Relationships 


Summary 

Training  effeetiveness  models  have  repeatedly  used  trainee  eharaeteristies, 
organization  eharaeteristies,  and  training  outcomes  to  measure  the  effectiveness  of 
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training.  While  variable  anteeedents  may  vary  from  study  to  study,  the  premise  is  the 
same;  trainee  eharacteristies,  organization  charaeteristies,  and  training  outcomes  may  be 
strong  indicators  of  training  effectiveness  and,  subsequently,  trainee  motivation  to 
transfer  learned  skills  to  the  work  environment. 

Kirkpatrick  (1977)  laid  the  ground  work  for  measuring  training  effectiveness  with 
his  four-level  model.  A  number  of  theoretical  models  are  available  to  guide  the 
measurement  of  training.  Specifically,  Tannenbaum  et  al.  (1991)  studied  the  effect  of 
training  characteristics  on  training  effectiveness  and  Mathieu  and  Martineau  (1997) 
incorporated  organization  characteristics  and  their  effect  on  training  effectiveness.  This 
study  integrates  the  trainee  characteristics,  organization  characteristics,  and  training 
outcomes  of  these  models  to  examine  the  relationship  between  said  constructs  and 
immediate  training  outcomes  in  support  of  Kirkpatrick’s  first  and  second  levels  of 
training  effectiveness  measurement:  reaction  and  learning.  Additionally,  this  study 
examines  the  relationship  between  said  constructs  and  the  possible  future  training 
outcome  of  the  trainee’s  motivation  to  transfer  learned  skills  to  the  work  environment. 

The  military  supports  an  iterative  instructional  systems  development  model  where 
evaluation  is  part  of  the  cyclical  training  process  (Department  of  the  Air  Force,  1993). 
Lack  of  evaluation  in  the  military  due  to  misconceptions  (Salas  et  ah,  2003)  or  confusion 
about  what  to  measure  (Alliger  &  Tannenbaum,  1996)  is  common.  BCOT  evaluation 
includes  immediate  trainee  reactions  and  post-training  task-related  self-efficacy 
measures.  This  study  attempts  to  develop  a  robust  training  effectiveness  model  for  use 
through  the  Air  Force  C&l  officer’s  training  career.  The  model  used  in  this  study 
integrates  previous  research  findings  and  posits  that  certain  trainee  characteristics  and 
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organization  characteristics  (i.e.,  affective  organization  eommitment,  task-related  self- 
effieaey,  learning  self-effieaey,  training  motivation,  organization  support,  organization 
eonstraints,  and  opportunity  to  perform  skills  learned)  are  related  to  training  outeomes  as 
measured  by  the  trainee’s  performanee,  reaction  to  training,  and  motivation  to  transfer 
skills  learned  to  the  job.  Furthermore,  training  performanee  and  reaetion  are  expeeted  to 
be  indieators  of  immediate  training  effeetiveness,  while  motivation  to  transfer  is  expeeted 
to  be  indicative  of  future  job  performance  whieh  may  or  may  not  be  a  further  indieation 
of  training  effectiveness. 

Relationships  between  these  construets  and  training  effeetiveness  may  provide 
insight  into  issues  that  may  be  hindering  training  effectiveness.  In  turn,  by  identifying 
possible  problems  and  potential  solutions  with  the  current  training  evaluation,  further 
development  of  future  Air  Foree  C&I  Offieers  through  training  may  be  possible.  The 
following  ehapter  will  outline  the  researeh  methodology  used  in  this  study.  Chapter  four 
will  detail  the  analysis  of  the  data,  and  chapter  five  will  discuss  the  researeh  findings, 
research  limitations,  and  recommendations  for  further  researeh  in  this  area. 
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III.  Methodology 


Overview 

The  previous  ehapters  outlined  the  eurrent  problem  statement  and  reviewed 
literature  pertaining  to  training  evaluation  and  training  effeetiveness.  Trainee 
eharaeteristies  and  organizations  characteristics  were  combined  into  one  model  to  further 
the  study  of  training  effectiveness.  This  chapter  outlines  the  methodology  used  to 
develop  and  deploy  the  training  effectiveness  surveys,  which  were  designed  to  measure 
the  research  questions  presented  in  chapter  one  and  the  hypotheses  presented  in  chapter 
two.  The  training  effectiveness  surveys  addressed  each  of  the  constructs  outlined  in  the 
proposed  research  model  (Figure  2).  This  chapter  covers  information  regarding  the 
following  areas;  samples  size;  survey  procedures;  development  and  reliability  of  the 
survey  instruments;  and  data  analysis  methods. 

Sample 

The  targeted  participants  for  this  study  were  Air  Force  Communication  and 
Information  (C&I)  Officers  attending  BCOT  whose  class  start  dates  were  between 
September  15,  2003  and  September  30,  2003.  Three  classes  with  a  total  of  49  students 
started  during  this  time  frame.  These  classes  provided  a  representative  sample  of  training 
students  that  attend  BCOT  throughout  the  year.  Students  were  company  grade  officers 
(i.e.,  second  lieutenants,  first  lieutenants,  or  captains),  contractors,  and  foreign  officers. 
Thirty-nine  useable  responses  were  obtained  from  the  sample.  Responses  that  did  not 
include  both  a  pre-training  survey  matched  with  a  post-training  survey  were  not  included 


42 


in  the  data  collection.  Reasons  for  unmatchable  surveys  included  students  being  unable 
to  finish  the  course  or  students  finishing  the  course  early  due  to  prior  commitments. 

Procedures 

During  the  first  day  of  BCOT  class,  the  researcher  was  allotted  time  to  conduct 
the  pre-training  survey.  The  researcher  explained  who  was  conducting  the  research,  what 
the  research  concerned,  why  the  students  had  been  targeted  as  participants,  and  how  the 
results  would  be  used.  She  also  explained  that  their  responses  would  be  collected 
anonymously  and  provided  contact  information.  At  this  point  in  their  training,  the 
trainees  were  asked  to  complete  a  pre-training  survey  (see  Appendix  A)  that  assessed 
affective  organization  commitment,  task-related  self-efficacy,  learning  self-efficacy,  and 
training  motivation.  Demographic  information  was  also  collected  as  part  of  the  pre¬ 
training  questionnaire.  During  the  final  week  of  attendance,  the  researcher  asked  the 
trainees  to  complete  a  post-training  survey  (see  Appendix  B)  that  assessed  organization 
support,  training  reaction,  motivation  to  transfer,  situational  (i.e.,  organization) 
constraints,  and  opportunity  to  perform.  Additionally,  affective  organization 
commitment  and  task-specific  self-efficacy  were  re-assessed.  The  re-assessment 
questions  were  randomly  reordered  to  ensure  participants  answers  were  not  memory 
based.  Each  survey  was  provided  with  an  instruction  set  and  completed  in  the  presence 
of  the  researcher  in  order  to  bolster  participation  and  to  provide  answers  to  any  questions 
that  may  have  arisen.  The  last  five  digits  of  the  subject’s  social  security  number  or 
student  ID  were  used  to  record  pre-training  and  post-training  survey  answers  as  well  as  to 
match  performance  records  with  appropriate  survey  responses. 
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Measures 


Except  where  otherwise  noted,  all  measurement  responses  were  given  using  a  7- 
point  Likert- type  seale  ranging  from  strongly  disagree  (1)  to  strongly  agree  (7)  with 
neither  agree  nor  disagree  (4)  as  the  midpoint.  Kraiger  et  al.  (1993)  noted  the  most 
eommon  method  to  measure  attitude  is  with  a  seale  that  allows  respondents  to  indieate 
preference  or  rejeetion  of  the  object  such  as  the  one  used.  Both  survey  instruments  for 
this  study  used  questions  taken  from  previously  validated  surveys  to  ensure  validity. 
Appendix  C  summarizes  the  original  questions;  the  researcher(s);  the  construct  definition; 
the  original  scale  measurement  used;  and  the  modified  versions  of  each  question  used  in 
this  study. 

Trainee  Characteristics. 

Given  the  eontext  of  this  study  and  the  extent  of  the  literature,  four  trainee 
attitudes  were  examined.  They  were  affective  organization  commitment,  training  self- 
effieacy,  learning  self-effieacy,  and  training  motivation. 

Affeetive  Organization  Commitment.  Affeetive  organization  commitment 
was  assessed  with  the  eight-item  self-report  affective  commitment  scale  (ACS)  developed 
by  Meyer  and  Allen  (1984).  Meyer  and  Allen  tested  the  scale  against  previously  used 
reliable,  valid  scales  (viz.,  the  Organizational  Commitment  Questionnaire,  the  Ritzer- 
Trice  Scale,  and  the  Hrebiniak-Alutto  Seale).  The  ACS  has  shown  acceptable  reliability 
in  researeh  studies.  Meyer  and  Allen,  in  two  separate  studies  (Allen  &  Meyer,  1990; 
1984),  administered  the  seale  and  reported  a  eoefficient  alpha  of  .87.  Tracey,  Hinkin, 
Tannenbaum,  and  Mathieu’s  (2001)  researeh  reported  a  eoeffieient  alpha  of  .75. 
Additionally,  Meyer  and  Allen  found  the  seale  correlated  highly  (r  =  .78)  with  the 
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Organizational  Commitment  Questionnaire.  This  eorrelation  provides  some  evidenee  of 
eonvergent  validity.  Construct  validity  was  shown  when  affective  commitment 
manipulation  had  the  expected  effect  on  all  the  scales  measuring  affective  commitment 
including  the  newly  developed  ACS.  Allen  and  Meyer’s  findings  of  correlation  between 
the  ACS  and  proposed  antecedent  variables,  specifically  skills,  provide  further  evidence 
of  validity.  Scale  items  were  modified  for  use  in  the  Air  Force  setting.  Four  items  were 
reverse-coded.  Sample  items  include  “The  Air  Force  has  a  great  deal  of  personal 
meaning  to  me”  and  “I  do  not  feel  ‘emotionally  attached’  to  the  Air  Force”  (reversed). 
Table  2  presents  the  affective  organization  commitment  scale  items. 


Table  2.  Affective  Organization  Commitment  Modified  Scale  Items 

Affective  Organization  Commitment  Scale _ 

1 .  I  would  be  very  happy  to  fulfill  a  eareer  in  the  Air  Foree. 

2.  I  enjoy  diseussing  the  Air  Foree  with  those  not  in  the  Air  Foree. 

3.  I  really  feel  as  if  the  Air  Foree's  problems  are  my  own. 

4.  I  think  that  I  eould  easily  beeome  as  attaehed  to  another  organization  as  I  am  to 
the  Air  Foree.  (R) 

5.  I  do  not  feel  like  'part  of  the  Air  Foree  family'.  (R) 

6.  I  do  not  feel  'emotionally  attaehed'  to  the  Air  Foree.  (R) 

7.  The  Air  Foree  has  a  great  deal  of  personal  meaning  to  me. 

8.  I  do  not  feel  a  strong  sense  of  belonging  to  the  Air  Foree.  (R) 

NOTE:  (R)  indieates  item  is  reverse  eoded. 


Task-related  Self-Efficacy.  Task-related  self-efficacy  was  assessed  with  a 
scale  modeled  after  Robertson  and  Sadri’s  (1993)  scale.  Robertson  and  Sadri  developed 
a  57-item  managerial  self-efficacy  scale.  They  divided  the  odd-  and  even-numbered 
items  into  two  versions  of  the  scale  during  a  pilot  test  and  achieved  a  coefficient  alpha  of 
.97  for  Version  A  and  an  coefficient  alpha  of  .96  for  Version  B.  Robertson  and  Sadri  also 
conducted  a  separate  validation  study  in  which  both  versions  were  re-administered  to  a 
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different  sample.  Both  versions  of  the  seale  showed  ability  to  predict  the  mean  job 
performance  rating. 

For  this  study,  eleven  items,  one  for  each  block  of  instruction,  were  developed  to 
measure  task-related  self-efficacy  of  BCOT  trainees.  For  example,  for  the  Resource 
Management  Block  of  instruction,  the  item  read  “When  making  my  best  effort,  I  would 
be  able  to  successfully  manage  an  Air  Force  communications  project.”  The  scale  was 
reviewed  by  two  individuals  familiar  with  the  BCOT  curriculum  to  provide  some 
validation  of  the  item  content.  After  the  review,  minor  adjustments  were  made  to  correct 
grammatical  errors.  The  questions  were  also  modified  so  that  respondents  would  be  able 
to  answer  using  a  7-point  Likert-type  scale.  Table  3  presents  the  task-related  self- 
efficacy  scale  items. 


Table  3.  Task-Related  Self-Efficacy  Modified  Scale  Items 


Task-Related  Self-Efficacy  Scale _ 

When  making  my  best  effort,  I  would  be  able  to  - 

1 .  -  identify  the  role  of  Cl  offieers  and  eiviban  professionals  within  the  framework  of 
the  Air  Foree  Mission. 

2.  -  identify  eurrent  eommunieations  systems  employed  to  support  the  US  Air  Foree 
mission. 

3.  -  diseuss  the  faeets  of  life  eyele  management  from  a  eommunieations  perspeetive. 

4.  -  explain  how  to  treat  Air  Foree  eommunieations  systems  as  weapons  systems. 

5.  -  discuss  the  facets  of  network  operating  systems  and  the  roles  and  responsibilities  of 
the  Network  Control  Center  (NCC). 

6.  -  configure  and  manage  a  UNIX-based  network  operating  system. 

7.  -  confidently  install,  configure,  and  manage  the  Windows  NT  operating  system. 

8.  -  successfully  manage  an  Air  Force  communications  project. 

9.  -  discuss  Cl  authority  and  responsibility  from  the  MAJCOM  level  down  to  the  Cl 
Systems  Officer  at  the  base  level,  including  deployed  operations  and  expeditionary 
aerospace  forces. 

10.  -  describe  the  Military  Satellite  Communication  systems  and  explain  how 
commercial  satellite  systems  impact  the  DOD. 

11.  -  plan  and  deploy  a  communications  network. _ 
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Learning  Self-Efficacy.  Learning  self-efficacy  was  assessed  using  a  scale 
developed  by  Pintrich  et  al.  (1993).  Their  scale  assessed  a  student’s  expectancy  for 
success  and  self-efficacy.  Expectancy  for  success  relates  to  task  performance,  while  self- 
efficacy  is  the  self-appraisal  of  one’s  ability  to  master  a  task.  Pintrich  et  al.  (1993) 
reported  a  coefficient  alpha  of  .93  for  their  scale.  Slight  modification  for  the  Air  Force 
training  environment  was  needed  for  these  items.  Table  4  presents  the  learning  self- 
efficacy  scale  items. 


Table  4.  Learning  Self-Efficacy  Modified  Scale  Items 

Learning  Self-Efficacy  Scale _ 

1. 1  believe  I  will  receive  an  excellent  grade  in  this  course. 

2.  I'm  confident  I  can  do  an  excellent  job  on  the  assignments  and 
tests  in  this  training  course. 

3. 1  expect  to  do  well  in  this  training. 

4.  Considering  the  difficulty  of  the  blocks,  the  instructors,  and  my 
skills,  I  think  I  will  do  well  in  this  course. 

5.  I'm  certain  I  can  understand  the  most  difficult  material  presentei 
in  the  readings  for  this  course. 

6.  I'm  confident  I  can  understand  the  basic  concepts  taught  in  this 
course. 

7.  I'm  confident  I  can  understand  the  most  complex  material 
presented  by  the  instructor  in  this  course. 

8.  I'm  certain  I  can  master  the  skills  being  taught  in  this  course. 


Training  Motivation.  Training  motivation  was  assessed  using  a  6-item 
seale  that  has  been  administered  in  two  previous  studies  (Warr,  Allan,  &  Bird!,  1999; 
Warr  &  Bunee,  1995).  Coefficient  alphas  .94  and  .76  were  achieved  for  these  studies. 
An  illustrative  item  is  “Generally,  I  am  enthusiastic  about  learning  new  things.”  The 
scale  was  originally  developed  by  Warr  and  Bunee  (1995).  Items  cover  perceived 
personal  gains  and  interest  in  the  material  being  taught.  Slight  modification  for  the  Air 
Force  environment  was  needed  for  these  items.  Table  5  presents  the  training  motivation 
scale  items. 


47 


Table  5.  Training  Motivation  Modified  Seale  Items 

Training  Motivation  Scale _ 

1 .  Generally,  I  am  enthusiastic  about  learning  new  things. 

2.  Generally,  I  prefer  to  keep  away  from  training  courses  (R). 

3.  Generally,  I  am  keen  to  take  up  any  learning  opportunity  offered  to  me. 

4.  I  am  keen  to  learn  more  about  the  subjects  covered  in  this  training. 

5.  I  expect  that  this  training  will  help  me  a  lot  in  the  future. 

6.  This  training  is  really  a  waste  of  time  (R). _ 

NOTE:  (R)  indicates  item  is  reverse  coded. 


Organization  Characteristics. 

Three  organization  eharaeteristics  were  identified  as  appropriate  measures  of 
training  effectiveness:  organization  support,  organization  constraints,  and  opportunity  to 
perform. 

Organization  Support.  Organization  support  was  assessed  using  a  5 -item 
scale  based  on  research  by  Tracey  et  al.  (2001).  In  Tracey  et  al.’s  study,  the  organization 
support  scale  was  part  of  a  larger  work  environment  measurement.  Individual  internal 
reliability  was  not  reported  for  the  organization  support  piece  of  the  scale;  however,  the 
entire  scale  achieved  a  coefficient  alpha  of  .86.  Tracey  (2003)  establishes  further 
construct  validity  for  the  scale  in  his  study  reporting  a  coefficient  alpha  of  .90  for  the 
organization  support  scale.  Items  used  in  this  scale  had  been  factor  analyzed  in  a 
previous  study  by  Tracey  et  al.  (1995).  An  example  item  is  as  follows:  “In  this  store, 
coworkers  encourage  each  other  to  use  new  knowledge  and  skills  on  the  job.”  Items  were 
modified  for  the  Air  Force  setting.  Table  6  presents  the  organization  support  scale  items. 
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Table  6.  Organization  Support  Modified  Seale  Items 


Organization  Support  Scale _ 

1 .  There  is  a  performanee  appraisal  system  that  ties  rewards  to  the  use  of  newly  aequired 
knowledge  and  skills. 

2.  The  Air  Foree  offers  exeellent  training  programs. 

3.  Cl  offieers  are  provided  with  resourees  neeessary  to  aequire  and  use  new  knowledge  and 
skills. 

4.  There  are  rewards  and  ineentives  for  aequiring  and  using  new  knowledge  and  skills  in  one's 
job. 

5.  The  Air  Foree  rewards  Cl  offieers  for  using  newly  aequired  knowledge  and  skills  on  the  job. 


Organizational  Constraints.  Organizational  constraints  were  assessed 
using  a  16-item  scale  used  in  Mathieu,  Martineau,  and  Tannenbaum’s  (1992)  study.  The 
sixteen  items  in  this  seale  measured  the  extent  to  whieh  employees  pereeived  they  would 
be  eonstrained  in  the  areas  of  information  sourees;  equipment  and  supplies;  authority  to 
eomplete  their  jobs;  and  time  to  eomplete  their  jobs.  These  items  were  modified  to 
measure  the  BOOT  trainee’s  expectation  of  adequate  equipment,  time,  and 
eneouragement  they  would  reeeive  at  their  duty  station  after  the  training.  Mathieu  et  al.’s 
scale  aehieved  a  coeffieient  alpha  of  .85  after  two  items  that  exhibited  low  average 
interitem  eorrelations  were  dropped.  An  example  of  an  item  read  “I  will  have  adequate 
equipment  (e.g.,  eomputers,  software)  for  performing  my  job.”  Table  7  presents  the 
organization  eonstraints  seale  items. 
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Table  7.  Organization  Constraints  Modified  Seale  Items 

Organization  Constraints  Scale _ 

1.  I  will  receive  adequate  information  from  other  sources  (e.g.,  co-workers,  departments,  outside 
companies  or  agencies,  etc.)  needed  to  perform  my  job  well. 

2.  I  will  have  adequate  equipment  (e.g.,  computers,  software)  for  performing  my  job. 

3.  I  will  have  adequate  supplies  (e.g.,  paper,  equipment  parts)  for  performing  my  job. 

4.  There  is  a  shortage  of  help  in  my  unit. 

5.  I  will  have  the  opportunity  to  receive  adequate  educational  and/or  training  experiences  necessary 
to  perform  my  job  well. 

6.  There  will  be  enough  time  available  to  complete  my  job  duties  as  assigned. 

7.  The  physical  aspects  of  my  unit  (e.g.,  space,  lighting,  etc.)  will  be  adequate. 

8.  My  job  duties  and  tasks  will  be  scheduled  in  an  efficient  manner. 

9.  I  will  have  sufficient  authority  to  complete  the  tasks  that  are  assigned  to  me. 

10.  The  operating  budget  in  my  unit  is  sufficient  to  cover  the  amount  of  work  produced  in  my  unit. 

1 1 .  Administrative  rules  or  policies  will  hinder  my  effectiveness  on  the  job. 

12.  I  will  receive  sufficient  forewarning  to  plan  my  work  activities. 

13.  My  supervisor  will  encourage  me  to  learn  new  skills  or  to  try  out  new  ideas. 

14.  My  co-workers  will  resist  new  ideas  or  the  use  of  new  work  procedures. 

15.  My  unit  has  prescribed  ways  of  doing  things  that  must  be  followed. 

16.  Time  will  be  made  available  to  me  in  order  to  practice  new  skills  or  to  experiment  with  different 

work  procedures. _ 


Opportunity  to  Perform.  Opportunity  to  perform  was  assessed  using  an  1 1 -item 
seale  modeled  after  Ford  et  al.’s  (1992)  seale.  Their  seale  was  developed  speeifieally  for 
the  Aerospaee  Ground  Equipment  (AGE)  Airman  Basie-in-Residenee  teehnieal  training 
eourse.  The  seale  was  developed  to  measure  the  breadth,  aetivity  level,  and  type  of  tasks 
for  the  AGE  eareer  field.  The  internal  reliability  for  this  seale  was  .74.  This  seale  was 
modified  to  represent  the  tasks  expeeted  to  be  performed  by  new  C&I  offieers.  Eord  et 
al.  (1992)  eontend  that  measurement  should  be  at  the  appropriate  level.  While  the  AGE 
survey  measured  tasks  learned,  the  BCOT  eourse  focuses  more  at  the  knowledge  level. 
Therefore,  measurement  items  were  developed  in  an  effort  to  measure  how  much 
knowledge  learned  in  BCOT  would  be  transferred  and  used  on  the  job.  Eor  example,  the 
item  for  measuring  the  knowledge  learned  about  project  management  skills  read  “I  will 
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have  to  opportunity  to  manage  an  Air  Foree  eommunieations  projeet.”  Table  8  presents 
the  opportunity  to  perform  seale  items. 


Table  8.  Opportunity  to  Perform  Modified  Seale  Items 

Opportunity  to  Perform  Scale _ 

I  will  have  the  opportunity  to  - 

1 .  -  work  as  a  Cl  offieer. 

2.  -  work  with  eurrent  eommunieations  systems  employed  to  support  the  US 

Air  Foree  mission. 

3.  -  perform  life  eyele  management  on  a  eommunieations  system. 

4.  -  develop  or  use  Air  Foree  eommunieations  systems  as  weapons 

systems. 

5.  -  work  with  network  operating  systems  and  the  Network  Control  Center 

(NCC). 

6.  -  eonfigure  and  manage  a  UNIX-based  network  operating  system. 

7.  -  install,  eonfigure,  and  manage  a  Windows  NT  operating  system. 

8.  -  manage  an  Air  Foree  eommunieations  projeet. 

9.  -  work  at  the  base  level  in  an  aerospaee  expeditionary  foree. 

10.  -  work  with  the  Military  Satellite  Communieation  system  and 

eommereial  satellite  systems. 

11.  -  plan  and  deploy  a  eommunieations  network. 


Training  Outcomes. 

Training  Performance.  Training  performance  was  assessed  using  final 
course  grades.  The  final  course  grade  is  an  average  of  the  grades  received  from  the 
eleven  blocks  of  instruction.  This  grade  reflects  the  trainees’  learning  of  the  academic 
material  covered  in  the  course.  As  mentioned  previously,  students  were  identified  by  the 
last  five  digits  of  their  social  security  numbers  or  their  student  ID  number.  This  measure 
was  coded  as  a  z-score  where  higher  scores  reflected  better  performance. 

Training  Reaction.  Training  reaction  was  assessed  using  a  9-item  scale 
developed  by  Warr  et  al.  (1999).  The  scale  was  developed  to  measure  three  distinct 
reaction  areas:  enjoyment,  perceived  usefulness,  and  perceived  difficulty.  Three  items 
are  used  to  measure  each  area.  Factor  analysis  was  undertaken  to  verily  the  three  factors 
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of  reaction.  Eiganvalues  above  1.00  were  present  for  each  factor  and  varimax  rotation 
identified  the  three  proposed  components.  Alpha  coefficients  of  internal  reliability  were 
.76,  .80,  and  .75,  respectively.  Sample  items  include  “This  course  was  extremely 
interesting”  for  enjoyment;  “This  course  closely  related  to  my  job  needs”  for  perceived 
usefulness;  and  “I  found  this  course  very  hard  to  follow”  for  perceived  difficulty.  Items 
on  this  scale  were  modified  for  the  Air  Force  setting.  Table  9  presents  the  training 
reaction  scale  items. 


Table  9:  Training  Reaction  Modified  Scale  Items 

Training  Reaction  Scale _ 

Enjoyment 

1.  I  really  enjoyed  this  course. 

2.  This  course  was  very  good  fun. 

3.  This  course  was  extremely  interesting. 

Perceived  usefulness 

1 .  This  course  was  very  relevant  to  my  job. 

2.  This  course  was  of  great  practical  value  to  me  for  my  job. 

3.  This  course  was  closely  related  to  my  job  needs. 

Perceived  difficulty 

1 .  I  found  this  course  very  hard  to  follow. 

2.  I  thought  this  course  was  a  tough  one. 

3.  I  found  this  course  difficult  to  understand. 


Motivation  to  Transfer.  Motivation  to  transfer  was  assessed  using  a  3- 
item  scale  developed  by  Warr  et  al.  (1999).  An  example  item  is  “1  feel  very  committed  to 
apply  what  1  have  learned  to  my  job.”  In  Warr  et  al.’s  research,  the  three  items  from  this 
scale  were  included  in  a  factor  analysis  with  nine  other  reaction  items.  When  extracting 
four  factors,  the  items  distributed  as  expected.  The  scale  achieved  a  coefficient  alpha  of 
.79.  Items  on  this  scale  were  modified  for  the  Air  Force  setting.  Table  10  presents  the 
motivation  to  transfer  scale  items. 
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Table  10:  Motivation  to  Transfer  Modified  Seale  Items 


Motivation  to  Transfer  Scale _ 

1 .  I  am  keen  to  apply  what  I  have  learned  in  this  course. 

2.  I  intend  to  use  what  I  have  learned  in  this  course. 

3 .  I  feel  very  committed  to  applying  what  I  have  learned  in  this 

course  to  my  job. _ 


Table  11  summarizes  the  reliability  analysis  statistics  (i.e.,  Cronbach’s  alpha, 
mean,  and  standard  deviation)  for  each  scale  used  in  this  study  (Leedy  &  Ormrod,  2001). 
All  scales  achieved  a  coefficient  alpha  above  .71.  Additionally,  a  combined  scale  was 
included  in  this  table.  Training  Reaction  -  Total  combines  the  three  separate  training 
reactions  measured.  Overall,  a  value  of  .70  is  considered  acceptable,  with  values  above 
.90  desirable  (Nunnally  &  Bernstein,  1994). 


Table  11.  Reliability  Analysis  Statistics  for  Measurement  Scales 


Scale 

Cronbach’s  Alpha 

Mean 

Standard 

Deviation 

Affective  Organization 
Commitment  (Time  1) 

.85 

5.34 

0.96 

Affective  Organization 
Commitment  (Time  2) 

.88 

5.38 

0.98 

Task-related  Self-efficacy 
(Time  1) 

.93 

4.59 

1.29 

Task-related  Self-efficacy 
(Time  2) 

.89 

5.29 

0.81 

Motivation  to  Learn 

.73 

5.85 

0.75 

Learning  Self-efficacy 

.91 

6.15 

0.69 

Training  Reaction  -  Perceived 
Usefulness 

.89 

4.56 

1.40 

Training  Reaction  -  Perceived 
Difficulty 

.81 

2.97 

1.30 

Training  Reaction  -  Enjoyment 

.91 

4.09 

1.77 

Training  Reaction  -  Total 

.73 

3.87 

0.95 

Organizational  Constraints 

.71 

4.71 

0.49 

Opportunity  to  Perform 

.77 

4.75 

0.74 

Organization  Support 

.75 

4.83 

0.80 

Motivation  to  Transfer 

.83 

5.61 

0.92 
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Instrument  Review 


As  part  of  the  development  process,  the  surveys  were  reviewed  by  six  C&I 
Officers  familiar  with  BOOT  to  ensure  clarity  of  wording  and  instructions.  Feedback 
from  the  review  resulted  in  minor  wording  adjustments. 

Data  Analysis 

Descriptive  Statistics. 

Descriptive  statistics  for  each  construct  will  be  reported.  This  will  provide 
frequency  distribution,  central  tendency  (i.e.,  the  mean),  and  variation  for  the  sample 
studied.  In  turn,  inferences  from  this  data  may  be  made  of  the  population  of  all  Basic 
Communications  Officer  Training  (BCOT)  students.  Additionally,  reporting  these 
statistics  may  further  the  generalizability  of  the  model  beyond  the  military  scope 
(Kachigan,  1991). 

Correlation  Analysis. 

A  correlation  matrix  will  be  presented  that  includes  all  variables  measured.  It 
should  be  noted  that  two  scales,  affective  organization  commitment  and  task-related  self- 
efficacy,  were  measured  twice  during  this  study.  Both  measurements  of  these  two 
variables  will  be  reported  in  the  correlation  matrix.  The  correlation  matrix  will  be 
reviewed  for  significant  positive  and  negative  relationships  between  each  variable. 
Specifically,  training  and  organization  characteristics  are  expected  to  positively  correlate 
with  training  outcomes.  This  review  may  lead  to  further  regression  analysis. 

Analysis  of  Variance  (ANOVA) 

ANOVA  will  be  used  to  analyze  the  relationships  between  the  pre-training  and 
post-training  measurements  of  affective  organization  commitment  and  task-related  self- 
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efficacy.  Kraiger  et  al.  (1993)  identified  that  pre-  and  post-training  measurement  of 
attitudes  that  indicate  a  change  during  training  may  signal  that  learning  has  occurred. 
Evidence  of  success  may  be  derived  from  mean  differences  between  pre-  and  post-tests. 

Regression  Analysis. 

Regression  analysis  will  be  used  to  provide  mathematical  equations  for  identified 
relationships  in  the  correlation  matrix  using  training  performance,  training  reactions,  and 
motivation  to  transfer  as  the  dependent  variables  and  affective  organization  commitment, 
task-related  self-efficacy,  learning  self-efficacy,  training  motivation,  organization 
support,  organization  constraints,  opportunity  to  perform,  and  trainee  demographics  as 
the  independent  variables.  For  example,  it  is  expected  that  task-related  self-efficacy, 
learning  self-efficacy,  and  training  motivation  will  be  positively  correlated  with  training 
performance.  If  this  is  supported  by  the  correlation  analysis,  a  regression  model  using  all 
three  independent  variables  will  be  posited. 

Summary 

This  chapter  outlined  the  methodology  used  in  developing  and  conducting  the 
training  effectiveness  surveys  used  in  this  research.  Both  surveys  were  conducted  in 
person  at  Keesler  Air  Force  Base.  The  sample  population  included  49  students,  from 
which  39  useable  responses  were  received.  After  the  data  were  collected,  they  were 
analyzed  using  correlation  analysis,  analysis  of  variance,  and  regression  analysis.  In  the 
following  chapter,  the  data  are  outlined  and  analyzed.  Chapter  five  presents  conclusions 
and  recommendations  for  the  overall  study. 
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IV.  Data  Analysis 


Overview 

The  previous  ehapters  outlined  the  eurrent  problem  statement,  reviewed  literature 
pertaining  to  training  evaluation  and  training  effeetiveness,  and  presented  the  researeh 
questions  and  hypotheses  tested  in  this  study.  In  addition,  chapter  three  outlined  the 
methodology  for  collecting  and  analyzing  data  and  outlined  each  of  the  nine  measures 
that  comprised  the  pre-training  and  post-training  surveys  used  in  this  study.  This  chapter 
summarizes  the  surveys  findings  and  presents  the  data  analysis. 

In  an  effort  to  answer  research  questions  2  and  3  (i.e..  What  is  the  relationship 
between  trainee  characteristics  and  training  outcomes?  What  is  the  relationship  between 
organization  characteristics  and  training  outcomes?)  presented  in  chapter  one,  several 
data  analysis  techniques  were  used.  To  review,  five  hypotheses  posited  relationships 
among  the  constructs  in  the  proposed  research  model.  These  hypotheses  are  summarized 
again  in  Table  12. 

First,  hypotheses  1  and  2,  directly  relating  to  research  questions  2  and  3,  were 
examined  using  correlation  analysis.  Hypotheses  3,  4,  and  5  were  posited  as  further 
relationships  that  may  support  and  provide  answers  to  research  questions  2  and  3.  They 
also  were  examined  using  correlation  analysis.  A  correlation  matrix  will  be  presented 
later  in  this  chapter.  Additionally,  further  analysis  for  hypothesis  4,  relationships 
between  pre-training  and  post-training  characteristics,  was  performed  using  analysis  of 
variance  (ANOVA).  After  correlation  analysis  and  ANOVA  were  performed,  the 
proposed  research  model  was  revised  to  support  the  findings  of  the  analysis  and  propose 
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a  more  feasible  training  effeetiveness  model.  Finally,  in  an  effort  to  support  predietive 
ability  of  the  revised  researeh  model  (Kaehigan,  1991),  regression  analysis  was  used  to 
propose  mathematieal  equations  for  training  effectiveness. 


Table  12.  Hypotheses  Summary 


Hypothesis 

Description 

Hypothesis  1 

Hypotheses  to  support  relationships  between 
trainee  characteristics  and  training  outcomes. 

Hypothesis  2 

Hypotheses  to  support  relationships  between 
organization  characteristics  and  training 
outcomes. 

Hypothesis  3 

Hypotheses  to  support  relationships  between 
trainee  demographics  and  trainee  characteristics. 

Hypothesis  4 

Hypotheses  to  support  relationships  between  pre¬ 
training  and  post- training  trainee  characteristics. 

Hypothesis  5 

Hypotheses  supporting  relationships  between 
training  outcomes  and  post-training  trainee 
characteristics. 

Descriptive  Statistics 

Table  13  outlines  the  demographics  of  all  participants  who  responded  to  the 
survey.  Demographic  information  collected  from  respondents  included  age,  gender,  rank, 
time  in  service,  prior  job  knowledge,  and  Major  Command  (MAJCOM)  to  which 
assigned.  As  seen  in  Table  13,  demographics  are  shown  for  thirty-nine  respondents.  For 
this  table,  age  was  recoded  into  three  groups:  less  than  25,  25  -  30,  and  over  30. 
Additionally,  time  in  service  was  recoded  into  four  groups  of  two  years  each,  that  is,  less 
than  24  months,  24  -  48  months,  48  -  72  months,  and  more  than  72  months. 
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Table  13.  Sample  Demographics  Statistics 


Characteristic 

n 

% 

Characteristic 

n 

% 

Age  (in  years) 

<25 

15 

38.5 

Gender 

Female 

1 

17.9 

25-30 

14 

35.9 

Male 

32 

82.1 

>30 

10 

25.6 

Rank 

2”‘*  Fieutenant 

37 

94.8 

Prior  Job  Knowledge 

Some 

5 

12.8 

F*  Fieutenant 

1 

2.6 

None 

34 

%12 

Captain 

1 

2.6 

MajCom  Assignment 

No  Answer 

6 

15.4 

Time  in  Service 

(in  months) 

<24 

18 

46.2 

ACC 

8 

20.5 

24-48 

3 

7.7 

AFTC 

5 

12.8 

48-72 

6 

15.4 

AFMC 

2 

5.1 

>72 

12 

30.8 

AFSPC 

2 

5.1 

AMC 

8 

20.5 

PACAF 

3 

7.7 

SPACE 

2 

5.1 

USAFE 

3 

7.7 

Correlation  Analysis 

Hypotheses  one  through  five  were  analyzed  using  correlation  analysis.  Table  14 
presents  the  means  (M),  standard  deviations  (SD),  and  observed  correlations  among  all 
variables  in  this  study.  All  thirty-nine  respondents  answered  both  the  pre-training  and 
post-training  surveys;  of  which,  thirty-eight  were  matched  with  a  performance  score  by 
using  the  last  five  digits  of  their  social  security  number  or  student  ID  (i.e.,  only  one  set  of 
surveys  did  not  have  a  corresponding  performance  score).  In  addition,  two  combined 
measurements  are  included  in  the  correlation  matrix:  training  reaction-total  (TR-TTL) 
and  training  effectiveness  (TNG-EFF).  TR-TTF  combines  the  three  reaction 
measurements  (perceived  usefulness  -  PU,  perceived  difficulty  -  PD,  and  enjoyment  -  F) 
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into  one  score.  TNG-EFF  combines  the  training  outcomes  (performance,  training 
reaction-total,  and  motivation  to  transfer)  into  a  single  score.  Significant  correlations  will 
be  described  as  each  hypothesis  is  analyzed. 
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Table  14.  Correlation  Matrix  for  Study  Measures 


Measure 

M 

SD 

Correlation  Coefficients 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

1.  Performance 

91.03 

5.36 

2.  Age 

26.97 

4.36 

.43** 

3.  Gender 

N/A 

N/A 

-.14 

-.08 

4.  Rank 

N/A 

N/A 

.25 

.33 

-.09 

5.  Time  in  Service 

48.26 

51.21 

.36 

.75** 

-.18 

.40* 

(in  months) 

6.  MAJCOM 

N/A 

N/A 

.36* 

.19 

.13 

-.14 

.09 

7.  Prior  Job  Knowledge 

N/A 

N/A 

.32* 

.20 

-.22 

.14 

.37* 

.04 

8.  TRSE -Pre 

4.59 

1.29 

.17 

.14 

.12 

.02 

.17 

.25 

-.17 

9.  AOC -Pre 

5.34 

0.96 

-.25 

-.01 

-.10 

-.10 

.21 

-.05 

-.06 

.06 

10.  TM-Pre 

5.87 

0.75 

-.26 

.11 

-.16 

.15 

.08 

-.05 

-.30 

.03 

.61** 

11.  LSE -Pre 

6.15 

0.69 

.20 

.35* 

-.14 

-.02 

.30 

.37* 

-.16 

.22 

.16 

.46** 

12.  TR-PU 

4.56 

1.40 

-.26 

-.01 

.05 

.05 

-.09 

-.01 

-.25 

-.15 

.20 

.33* 

-.09 

13.  TR-PD 

2.97 

1.30 

-.51** 

-.20 

.37* 

-.09 

-.22 

-.36* 

-.27 

.03 

-.03 

.03 

-.31 

.07 

14.  TR-E 

4.09 

1.77 

-.18 

.22 

-.21 

.11 

.24 

.01 

-.14 

.13 

4]^** 

.10 

47*H= 

-.28 

15.  TR-TTL 

3.87 

0.95 

_  47** 

.04 

.06 

.06 

.00 

-.16 

-.33* 

.02 

.36* 

.43** 

-.13 

.82 

.32 

16.  OC-Post 

4.73 

0.49 

.19 

.20 

-.07 

.24 

.17 

.02 

-.08 

.10 

-.06 

.14 

-.09 

.22 

-.08 

17.  AOC-Post 

5.38 

0.98 

-.23 

.11 

-.22 

.06 

.23 

-.07 

-.07 

.08 

7]^** 

.63** 

.07 

.39* 

-.17 

18.  OTP-Post 

4.80 

0.74 

-.12 

.05 

-.06 

-.07 

.09 

-.01 

-.39* 

.17 

.12 

.34* 

.13 

.56* 

.15 

19.  TRSE-Post 

5.29 

0.81 

-.02 

.14 

-.04 

.14 

.05 

.08 

-.20 

.23 

.20 

.35* 

.24 

.32* 

-.24 

20.  MTT -Post 

5.61 

0.92 

-.20 

.15 

-.25 

.02 

.14 

.08 

-.17 

.01 

.56* 

.60* 

.26 

.58** 

-.25 

21.  OS-Post 

4.82 

0.80 

-.40* 

-.01 

-.29 

.24 

.10 

-.30 

-.11 

-.07 

.22 

.31 

-.18 

44** 

-.03 

22.  TNG-EFF 

3.20 

0.61 

.23 

.43** 

-.13 

.40* 

.05 

-.11 

.17 

.26 

.45** 

.17 

.60** 

-.25 

Notes:  a)  **  Correlation  is  significant  at  the  0.01  level  (2 -tailed),  b)  *  Correlation  is  significant  at  the  0.05  level  (2 -tailed),  c)  TRSE  -  Task-related  self- 
efficacy,  AOC  -  Affective  organization  commitment,  TM  -  Training  motivation,  LSE  -  Learning  self-efficacy,  TR-PU  -  Training  reaction-Perceived 
usefulness,  TR-PD  -  Training  reaction-Perceived  difficulty,  TR-E  -  Training  reaction-Enjoyment,  TR-TTL  -  Combined  training  reaction  score,  OC  - 
Organizational  constraints,  OTP  -  Opportunity  to  perform,  MTT  -  Motivation  to  transfer,  OS  -  Organization  Support,  TNG-EFF  -  Training  Effectiveness, 
Pre  -  Pre-training  measure.  Post-  Post-training  measure 


Table  14  (cont’d).  Correlation  Matrix  for  Study  Measures 


Measure  Correlation  Coefficients 


14 

15 

16 

17 

18 

19 

20 

15. 

TR-TTL 

.72 

16. 

OC-Post 

.21 

.20 

17. 

AOC-Post 

.62** 

.50** 

.16 

18. 

OTP-Post 

.43** 

.61** 

.36* 

.27 

19. 

TRSE-Post 

.54** 

.38* 

.40* 

.36* 

.31 

20. 

MTT  -  Post 

.62** 

.18 

.63** 

.32* 

.46** 

21. 

OS-Post 

.58** 

.57** 

.46** 

47** 

.40* 

22. 

TNG-EFF 

.61** 

.37* 

.45** 

.42** 

47** 

.75** 

Hypothesis  1  Analysis 

Hypothesis  1  stated  that  there  would  be  positive  relationships  between  pre¬ 
training  trainee  eharaeteristies  (affeetive  organization  eommitment  -  AOC,  task-related 
self-efficaey  -  TRSE,  learning  self-effieaey  -  LSE,  and  training  motivation  -  TM)  and 
training  outeomes  (performanee,  training  reaetion  -  total  -  TR-TTL,  and  motivation  to 
transfer  -  MET).  Twelve  relationship  predictions  were  made  (see  Table  15  for  results 
summary).  With  respect  to  the  proposed  research  model,  four  were  supported:  Hie 
stated  a  positive  relationship  between  AOC-Pre  and  TR-TTE  (training  reaction- 
enjoyment  was  also  positively  correlated  with  AOC-Pre);  Hlh  stated  a  positive 
relationship  between  TM  and  TR-TTL  (training  reaction-perceived  usefulness  and 
training  reaction-enjoyment  were  also  positively  correlated  with  TM);  Hli  stated  a 
positive  relationship  between  AOC-Pre  and  MTT;  and  Hll  stated  a  positive  relationship 
between  TM  and  MTT.  Therefore,  positive  relationships  between  all  pre-training 
trainee  characteristics  and  performance  were  not  supported.  In  addition,  positive 
relationships  between  the  pre-training  trainee  characteristics  of  TRSE  and  LSE  and 
training  outcomes  of  TR-TTL  and  MTT  were  not  supported.  However,  pre -training  AOC 

and  TM  were  positively  related  to  the  training  outcomes  of  TR-TTL  (as  well  as  training 
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reaction-perceived  usefulness  and  training  reaction-enjoyment)  and  MTT.  Overall,  only 
one-third  of  the  proposed  predictions  were  supported  for  hypothesis  1 . 


Table  15.  Hypothesis  1  Results  Summary 


Hypothesis 

Predicted  Relationship 

Result 

Hla 

As  AOC-Pre  +,  Performance  + 

Not  Supported 

Hlb 

As  TRSE-Pre  +,  Performance  + 

Not  Supported 

Hie 

As  LSE  +,  Performance  + 

Not  Supported 

Hid 

As  TM  +,  Performance  + 

Not  Supported 

Hie 

As  AOC-Pre  +,  TR  + 

Supported  (E) 

Hlf 

As  TRSE-Pre  +,  TR  + 

Not  Supported 

Hlg 

As  ESE  +,  TR  + 

Not  Supported 

Hlh 

As  TM  +,  TR  + 

Supported  (PU,  E) 

Hli 

As  AOC-Pre  +,  MTT  + 

Supported 

Hlj 

As  TRSE-Pre  +,  MTT  + 

Not  Supported 

Hlk 

As  ESE  +,  MTT  + 

Not  Supported 

Hll 

As  TM  +,  MTT  + 

Supported 

Hypothesis  2  Analysis 

Hypothesis  2  stated  that  there  would  be  certain  relationships  between  organization 
characteristics  (organization  support  -  OS,  organization  constraints  -  OC,  and  opportunity 
to  perform  -  OTP)  and  training  outcomes  (performance,  training  reaction-total  -  TR-TTL, 
and  motivation  to  transfer  -  MTT).  Nine  relationship  predictions  were  made  (see  Table 
16  for  results  summary).  With  respect  to  the  proposed  research  model,  four  were 
supported:  H2d  stated  a  positive  relationship  between  OS  and  TR-TTL  (training 
reaction-perceived  usefulness  and  training  reaction-enjoyment  were  also  positively 
correlated  with  OS);  H2f  stated  a  positive  relationship  between  OTP  and  TR-TTL 
(training  reaction-perceived  usefulness  and  training  reaction-enjoyment  were  also 
positively  correlated  with  OTP);  H2g  stated  a  positive  relationship  between  OS  and 
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MTT;  and  H2i  stated  a  positive  relationship  between  OTP  and  MTT.  Therefore,  none  of 
the  relationships  between  organization  eharacteristies  and  performance  were  supported. 

In  fact,  analysis  for  hypothesis  H2a,  originally  predicted  as  having  a  positive  relationship 
between  OS  and  performance,  supported  a  negative  correlation.  Additionally,  hypotheses 
predicting  negative  correlations  between  OC  and  training  outcomes  (TR-TTL  and  MTT) 
were  not  supported.  However,  OS  and  OTP  were  positively  related  to  TR-TTL 
(including  training  reaction-perceived  usefulness  and  training  reaction-enjoyment)  and 
MTT.  Overall  just  under  half  of  the  proposed  predictions  were  supported  for  hypothesis 
2. 


Table  16.  Hypothesis  2  Results  Summary 


Hypothesis 

Predicted  Relationship 

Results 

H2a 

As  OS  +,  Performance  + 

Opposite 

H2b 

As  OC  +,  Performance  - 

Not  Supported 

H2c 

As  OTP  +,  Performance  + 

Not  Supported 

H2d 

As  OS  +,  TR  + 

Supported  (PU,  E) 

H2e 

As  OC  +,  TR  - 

Not  Supported 

H2f 

As  OTP  +,  TR  + 

Supported  (PU,  E) 

H2g 

As  OS+,  MTT  + 

Supported 

H2h 

As  OC  +,  MTT  - 

Not  Supported 

H2i 

As  OTP  +,  MTT  + 

Supported 

Hypothesis  3  Analysis 

Hypothesis  3  stated  that  there  would  be  certain  relationships  between  trainee 
demographics  (age,  gender,  MAJCOM  Assignment,  Time  in  Service,  and  Prior  Job 
Knowledge)  and  pre-training  trainee  characteristics  (affective  organization  commitment  - 
AOC-Pre,  task-related  self-efficacy  -  TRSE-Pre,  learning  self-efficacy  -  LSE,  and 
training  motivation  -  TM).  Twenty  relationship  predictions  were  made  (see  Table  17  for 
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results  summary).  With  respect  to  the  proposed  research  model,  two  were  supported: 

H3k  stated  a  positive  relationship  between  age  and  LSE  and  H3m  stated  a  relationship 
between  MAJCOM  and  LSE.  Therefore,  none  of  the  relationships  between  AOC-Pre, 
TRSE-Pre,  and  TM  and  trainee  demographics  were  supported.  Additionally,  no 
relationship  between  gender  and  LSE  was  determined;  and  positive  relationships  between 
Time  in  Service,  Prior  Job  Knowledge  and  LSE  were  not  supported.  However,  age  was 
positively  related  to  LSE  and  MAJCOM  was  related  to  LSE.  Overall  only  two  of  the 
proposed  predictions  were  supported  for  hypothesis  3. 


Table  17.  Hypothesis  3  Results  Summary 


Hypothesis 

Predicted  Relationship 

Results 

H3a 

As  Age  +,  AOC-Pre  + 

Not  Supported 

H3b 

Gender  ?,  AOC-Pre  ? 

Not  Supported 

H3c 

MAJCOM  ?,  AOC-Pre  ? 

Not  Supported 

H3d 

As  Time  in  Service  +,  AOC-Pre  + 

Not  Supported 

H3e 

As  Prior  Job  Knowledge  +,  AOC-Pre  + 

Not  Supported 

H3f 

As  Age  +,  TRSE-Pre  + 

Not  Supported 

H3g 

Gender  ?,  TRSE-Pre  ? 

Not  Supported 

H3h 

MAJCOM  ?,  TRSE-Pre  ? 

Not  Supported 

H3i 

As  Time  in  Service  +,  TRSE-Pre  + 

Not  Supported 

H3j 

As  Prior  Job  Knowledge  +,  TRSE-Pre  + 

Not  Supported 

H3k 

As  Age  +,  LSE  + 

Supported 

H31 

Gender  ?,  LSE  ? 

Not  Supported 

H3m 

MAJCOM  ?,  LSE  ? 

Supported 

H3n 

As  Time  in  Service  +,  LSE  + 

Not  Supported 

H3o 

As  Prior  Job  Knowledge  +,  LSE  + 

Not  Supported 

H3p 

As  Age  -,  TM  + 

Not  Supported 

H3q 

Gender  ?,  TM  ? 

Not  Supported 

H3r 

MAJCOM  ?,  TM  ? 

Not  Supported 

H3s 

As  Time  in  Service  +,  TM  + 

Not  Supported 

H3t 

As  Prior  Job  Knowledge  +,  TM  + 

Not  Supported 
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Hypothesis  4  Analysis 

Hypothesis  4  stated  there  would  be  an  inerease  between  pre-training  and  post¬ 
training  measures  of  affeetive  organization  commitment  (AOC)  and  task-related  self- 
efficacy  (TRSE).  Based  on  correlation  analysis,  high  correlation  existed  between  pre- 
and  post-training  measures  of  AOC,  indicating  no  change  or  a  slight  change  between  the 
two  measurements.  On  the  other  hand,  low  correlation  existed  between  pre-  and  post¬ 
training  measures  of  TRSE,  indicating  a  signification  change  between  measurements. 
Eurther  support  for  these  results  is  presented  below  in  the  analysis  of  variance  section. 
Table  18  presents  the  summary  of  results  for  hypothesis  4. 


Table  18.  Hypothesis  4  Results  Summary 


Hypothesis 

Predicted  Relationship 

Results 

H4a 

AOC  will  increase  from  T1  to  T2 

Not  Supported 

H4b 

TRSE  will  increase  from  T1  to  T2 

Supported 

Hypothesis  5  Analysis 

Hypothesis  5  stated  that  there  would  be  positive  relationships  between  training 
outcomes  (performance,  training  reaction-total  -  TR-TTE  and  motivation  to  transfer  - 
MTT)  and  post-training  trainee  characteristics  (task-related  self-efficacy  -  TRSE-Post 
and  affective  organization  commitment  -  AOC-Post).  Six  relationship  predictions  were 
made  (see  Table  19  for  results  summary).  With  respect  to  the  proposed  research  model, 
four  were  supported;  H5b  stated  a  positive  relationship  between  TR-TTE  and  AOC-Post 
(training  reaction-perceived  usefulness  and  training  reaction-enjoyment  were  also 
positively  correlated  with  AOC-Post);  H5c  stated  a  positive  relationship  between  MTT 
and  AOC-Post;  H5e  stated  a  positive  relationship  between  TR-TTE  and  TRSE-Post 
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(training  reaction-perceived  usefulness  and  training  reaction-enjoyment  were  also 
positively  correlated  with  TRSE-Post);  and  H5f  stated  a  positive  relationship  between 
MTT  and  TRSE-Post.  Therefore,  none  of  the  relationships  between  performance  and 
post-training  trainee  characteristics  were  supported.  However,  TR-TTE  (including 
training  reaction-perceived  usefulness  and  training  reaction-enjoyment)  and  MTT  were 
positively  related  to  AOC-Post  and  TRSE-Post.  Overall  two-thirds  of  the  proposed 
predictions  were  supported  for  hypothesis  5. 


Table  19.  Hypothesis  5  Results  Summary 


Hypothesis 

Predicted  Relationship 

Results 

H5a 

As  Performance  +,  AOC-Post  + 

Not  Supported 

H5b 

As  TR  +,  AOC-Post  + 

Supported  (PU,  E) 

H5c 

As  MTT  +,  AOC-Post  + 

Supported 

H5d 

As  Performance  +,  TRSE-Post  + 

Not  Supported 

H5e 

As  TR  +,  TRSE-Post  + 

Supported  (PU,  E) 

H5f 

As  MTT  +,  TRSE-Post  + 

Supported 

Analysis  of  Variance  (ANOVA) 

Hypothesis  four  was  also  analyzed  using  analysis  of  variance  (ANOVA).  Table 
20  summarizes  the  statistics  and  tests  for  the  paired  samples  between  pre-  and  post¬ 
training  measures  of  task-related  self-efficacy  (TRSE)  and  affective  organization 
commitment  (AOC).  TRSE  means  changed  significantly  between  pre-  and  post-training 
measures;  while  AOC  means  did  not.  This  test  further  supports  the  correlation  analysis 
conducted  earlier. 
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Table  20.  Paired  Samples  Statisties  and  Test  Results 


Sample 

Mean 

SD 

t 

df 

Sig.  (2-tailed) 

Pair 

TRSE-Pre 

4.59 

1.29 

-3.22 

38 

.003 

1 

TRSE-Post 

5.29 

0.81 

Pair 

AOC-Pre 

5.34 

0.96 

-.32 

38 

.748 

2 

AOC-Post 

5.38 

0.98 

Other  Significant  Relationships 

This  study  measured  22  variables  whieh  provided  23 1  possible  eorrelations.  The 
proposed  research  model  identified  possible  relationships  for  76  of  the  231  possible 
correlations.  With  regard  to  the  proposed  research  model,  of  those  76  correlations,  26 
were  supported,  1  was  incorrectly  posited,  and  49  were  not  supported.  Further  study  of 
the  correlation  matrix  identified  42  other  significant  correlations  of  the  155  possible 
correlations  not  identified  by  the  proposed  research  model.  Table  21  summarizes  these 
42  significant  correlations.  Given  the  magnitude  of  this  finding,  a  revised  research  model 
is  presented. 
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Table  21 .  Other  Identified  Study  Variable  Relationships 


Relationship 

Correlation 

Relationship 

Correlation 

Performance  with  Age 

Positive 

TM  with  TNG-EFF 

Positive 

Performance  with  MAJCOM 

Positive 

TR-PU  with  TR-E 

Positive 

Performance  with  PJK 

Positive 

TR-PU  with  MTT 

Positive 

Performance  with  TR-PD 

Negative 

TR-PU  with  TNG-EFF 

Positive 

Performance  with  TR-TTL 

Negative 

TR-E  with  MTT 

Positive 

Age  with  Time  in  Service 

Positive 

TR-E  with  TNG-EFF 

Positive 

Age  with  TNG-EFF 

Positive 

TR-TTL  with  MTT 

Positive 

Gender  with  TR-PD 

Positive  (Males 

TR-TTL  with  TNG-EFF 

Positive 

perceived  course  more 

difficult) 

Rank  with  Time  in  Service 

Positive 

OC-Post  with  OTP 

Positive 

Rank  with  TNG-EFF 

Positive 

OC-Post  with  TRSE-Post 

Positive 

Time  in  Service  with  PJK 

Positive 

OC-Post  with  OS 

Positive 

Time  in  Service  with  TNG-EFF 

Positive 

OC-Post  with  TNG-EFF 

Positive 

MAJCOM  with  TR-PD 

Negative 

AOC-Post  with  TRSE-Post 

Positive 

PJK  with  TR-TTL 

Negative 

AOC-Post  with  OS 

Positive 

PJK  with  OTP 

Negative 

AOC-Post  with  TNG-EFF 

Positive 

AOC-Pre  with  TM-Pre 

Positive 

OTP  with  OS 

Positive 

AOC-Pre  with  AOC-  Post 

Positive 

OTP  with  TNG-EFF 

Positive 

TM  with  LSE 

Positive 

TRSE-Post  with  OS 

Positive 

TM  with  AOC-Post 

Positive 

TRSE-Post  with  TNG-EFF 

Positive 

TM  with  OTP 

Positive 

MTT  with  TNG-EFF 

Positive 

TM  with  TRSE-Post 

Positive 

OS  with  TNG-EFF 

Positive 

Revised  Research  Model 

The  revised  researeh  model  is  presented  in  Figure  7.  The  revised  model  reflects 
several  observations  based  on  analysis  of  the  original  proposed  research  model.  Trainee 
demographic  information,  while  important  to  gain  insight  of  the  sample  population,  did 
not  lend  itself  as  an  important  training  effectiveness  variable  to  the  original  model  for 
various  reasons.  There  was  a  lack  of  significant  correlations  with  other  variables 
identified  by  the  proposed  research  model.  Gender  had  only  one  significant  relationship 
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Pre-Training  Trainee  Characteristics 


Task- related 


Figure  7.  Revised  Research  Model 


which  was  with  training-perceived  difficulty.  This  may  be  because  of  the  limited  number 
of  survey  respondents.  The  age  demographic  relationships  did  not  lend  themselves  to 
further  the  study  of  training  effectiveness,  either.  These  relationships  included:  a)  time 
in  service  positively  related  with  age  (expected);  b)  performance  positively  related  to  age 
(unexpected);  and  c)  learning  self-efficacy  positively  related  to  age  (expected).  Rank  was 
discarded  from  the  revised  model  because  there  was  only  one  significant  correlation: 
rank  was  positively  correlated  with  time  in  service  (expected).  This  may  be  because  there 
were  only  two  students  who  ranked  higher  than  second  lieutenant  (i.e.,  one  first  lieutenant 
and  one  captain)  in  the  sample.  MAJCOM  assignment  was  discarded  from  the  revised 
model  because  of  the  large  number  of  second  lieutenants  identified  in  the  course.  The 
three  significant  correlations  identified  had  no  meaning  due  to  the  lack  of  experience  of 
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the  students  at  their  assignment  stations.  Prior  job  knowledge  was  discarded  for  the  same 
reason  as  MAJCOM  assignment.  Finally,  time  in  service  was  discarded  because,  like 
age,  it  did  not  lend  itself  to  further  the  study.  The  three  significant  correlations  identified 
with  time  in  service  were  as  expected,  two  of  which  were  with  other  trainee  demographic 
information. 

Pre-training  trainee  characteristics  were  also  modified.  Due  to  the  non-significant 
change  in  means  from  pre-  to  post-training  measurement,  affective  organization 
commitment  does  not  need  to  be  measured  twice.  Therefore,  it  was  removed  from  the 
pre-training  trainee  characteristics  list.  Further,  while  the  pre-training  measure  of  task- 
related  self-efficacy  did  not  correlate  with  any  other  measures,  it  was  retained  as  an 
important  measurement  to  determine  a  change  in  the  pre-  and  post-training  task-related 
self-efficacy  measurement.  Learning  self-efficacy  was  also  retained  although  it  was 
identified  to  only  correlate  with  motivation  to  learn  (i.e.,  training  motivation-TM).  A 
final  change  for  pre -training  trainee  characteristics  in  the  revised  model  is  the  proposed 
correlations  with  organization  characteristics  in  addition  to  training  outcomes. 

Training  outcomes  were  modified  to  include  only  training  reactions-total  and 
motivation  to  transfer.  Performance  correlated  with  three  trainee  demographics  (i.e.. 

Age,  MAJCOM,  and  PJK)  and  only  two  other  characteristics  in  this  study  of  training 
effectiveness  and  was  removed  from  the  revised  model.  Additionally,  motivation  to 
transfer  was  retained  in  the  training  outcomes  section  and  removed  from  the  post-training 
trainee  characteristics  list. 

Finally,  the  training  effectiveness  (TNG-EFF)  variable,  a  mean  combined  score  of 
training  reaction-total  and  motivation  to  transfer,  is  included  in  the  revised  model  as  it 
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had  significant  correlations  with  almost  every  variable  studied  in  both  models.  Further, 
removing  performance  from  the  training  effeetiveness  variable  equation  eaused  the  mean 
for  the  variable  to  increase  and  the  data  to  beeome  more  normally  distributed  (determined 
by  an  analysis  of  skewness  and  kurtosis).  Table  24  provides  a  comparison  of  the  training 
effeetiveness  variable  for  the  original  and  revised  models. 


Table  22.  Training  Effectiveness  Variable  Statistics 


TNG-EFF  (Original  Model) 

TNG-EFF  (Revised  Model) 

N 

39 

39 

Mean 

3.20 

4.74 

SD 

0.61 

0.83 

Varianee 

0.37 

0.70 

Skewness 

0.58 

-0.17 

Std.  Error  of  Skewness 

0.38 

0.38 

Kurtosis 

-0.80 

-0.15 

Std.  Error  of  Kurtosis 

0.74 

0.74 

The  revised  researeh  model  is  presented  as  a  simplifieation  of  the  original 
proposed  researeh  model  and  provides  significant  relationships  identified  from  research 
of  the  original  proposed  model.  Of  the  forty-five  possible  eorrelations  in  the  revised 
model,  twenty-nine  showed  signifieant  findings.  See  table  25  for  the  eorrelation  matrix 
of  the  revised  researeh  model. 
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Table  23.  Revised  Model  Correlation  Matrix 


Variable 

M 

SD 

1 

2 

3 

Correlation  Coeffieient 

4  5  6  7 

8 

9  10 

1.  TM 

5.87 

0.75 

2.  LSE 

6.15 

0.69 

.46** 

3.  TR 

3.87 

0.95 

.43** 

-.13 

4.  OC 

4.73 

0.49 

.14 

-.09 

.20 

5.  AOC 

5.38 

0.98 

.63** 

.07 

.50** 

.16 

6.  OTP 

4.80 

0.74 

.34* 

.13 

.61** 

.36* 

.27 

7.  TRSE 

5.29 

0.81 

.35* 

.24 

.28* 

.40* 

.36* 

.31 

8.  MTT 

5.61 

0.92 

.60** 

.26 

.62** 

.18 

.63** 

.32* 

.46** 

9.  OS 

4.82 

0.80 

.31 

-.18 

.57* 

47** 

.40* 

10.  TNG-EFF 

4.74 

0.84 

.58** 

.07 

.21 

.62** 

.52** 

47** 

9Q** 

.54** 

NOTE:  Notes:  a)  **  Correlation  is  significant  at  the  0.01  level  (2 -tailed),  b)  *  Correlation  is  significant 
at  the  0.05  level  (2 -tailed),  c)  TM  -  Training  motivation,  LSE  -  Learning  self-efficacy,  TR  -  Training 


reaction,  OC  -  Organizational  constraints,  AOC  -  Affective  organization  commitment,  OTP  -  Opportunity 
to  perform,  TRSE  -  Task-related  self-efficacy,  MTT  -  Motivation  to  transfer,  OS  -  Organization  Support, 
TNG-EFF  -  Training  Effectiveness 


Regression  Analysis 

According  to  Kachigan  (1991),  a  mathematical  regression  equation  may  be  used 
to  predict  future  outcomes  and  help  in  understanding  correlation  analysis.  Using  the 
revised  researeh  model  and  stepwise  regression  analysis,  three  regression  equations  are 
proposed  for  future  research  to  support  the  revised  research  model.  The  three  dependent 
variables  used  in  the  equations  were  the  training  outcomes  (motivation  to  transfer  and 
training  reaction-total)  and  the  training  effectiveness  variable.  Each  regression  analysis 
started  with  all  seven  independent  variables:  affective  organization  commitment, 
learning  self-efficacy,  task-related  self-efficacy,  training  motivation,  organization 
constraints,  opportunity  to  perform,  and  organization  support. 

The  first  regression  analysis  (Model  1)  was  performed  for  the  dependant  variable 
motivation  to  transfer.  Three  predictors  (affective  organization  commitment,  task-related 

self-efficacy,  and  training  motivation)  provide  an  R  of  .51  (adjusted  R  =  .46).  Adding 

2  2 

additional  predietors  raised  R  to  .52  (adjusted  R  =  .41).  The  seeond  regression  analysis 
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(Model  2)  was  performed  for  the  dependant  variable  training  reaction-total.  Three 
predictors  (learning  self-efficacy,  opportunity  to  perform,  and  training  motivation) 
provide  an  of  .55  (adjusted  =  .42).  Adding  additional  predictors  raised  R^  to  .64 
(adjusted  R  =  .56).  The  third  regression  analysis  (Model  3)  was  performed  for  the 
dependant  variable  training  effectiveness.  Two  predictors  (opportunity  to  perform  and 
affective  organization  commitment)  provide  an  R  of  .52  (adjusted  R  =.50).  Adding 
additional  predictors  raised  R  to  .62  (adjusted  R  =  .53).  A  summary  of  the  proposed 
regression  models  is  shown  in  Table  24.  Both  unstandardized  and  standardized  Beta 
weights  are  presented  for  each  model. 


Table  24.  Regression  Model  Summary 


Model 

R^ 

Adjusted 

R^ 

Unstandardized 

Coefficients 

Std. 

Beta  Error 

Standardized 

Coefficients 

Beta 

t 

Sig. 

1  for 

MTT 

.51 

.46 

(Constant) 

0.31 

0.97 

0.32 

.75 

TM 

0.36 

0.19 

0.30 

1.91 

.07 

AOC 

0.34 

0.15 

0.36 

2.30 

.03 

TRSE 

0.26 

0.15 

0.23 

1.78 

.08 

2  for 

TR 

.55 

.52 

(Constant) 

0.83 

1.15 

0.72 

.48 

LSE 

-0.55 

0.18 

-0.40 

-3.12 

.00 

TM 

0.56 

0.17 

0.44 

3.29 

.00 

OTP 

0.65 

0.15 

0.51 

4.26 

.00 

3  for 

TNG-EEE 

.52 

.50 

(Constant) 

0.27 

0.74 

0.37 

.71 

AOC 

0.45 

0.10 

0.52 

4.36 

.00 

OTP 

0.43 

0.14 

0.38 

3.20 

.00 
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Summary 

This  chapter  outlined  the  results  obtained  during  this  study.  Correlation  analysis, 
ANOVA,  and  regression  analysis  were  used  to  analyze  the  data  eollected.  Correlation 
analysis  was  used  for  all  five  hypotheses  to  determine  significant  relationships  between 
variables  presented  in  the  original  proposed  researeh  model.  Few  significant 
relationships  were  found.  Further,  ANOVA  was  used  to  examine  hypothesis  four  and 
provided  further  support  for  the  relationships  found  between  the  pre-  and  post-training 
measures  of  affective  organization  commitment  and  task-related  self-effieaey  in  the 
correlation  analysis.  Given  the  minimal  significant  relationships  identified  from  the 
original  model,  a  revised  model  was  proposed  and  analyzed  using  correlation  analysis. 

Of  the  45  possible  eorrelations  for  the  revised  model,  almost  65%  (i.e.,  29  correlations) 
were  significant.  With  the  revised  model  as  a  baseline,  stepwise  regression  analysis  was 
used  to  propose  mathematical  equations  for  determining  the  three  training  outcomes 
(motivation  to  transfer,  training  reaetion-total,  and  training  effeetiveness)  that  were 
retained.  The  following  ehapter  will  provide  conelusions  and  recommendations  based  on 
the  results  presented  in  this  chapter. 
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V.  Conclusion 


Overview 

The  overall  purpose  of  this  study  was  to  investigate  the  relationship  between 
training  effeetiveness  and  trainee  and  organization  eharaeteristies  using  the  BCOT  eourse 
as  a  test  bed.  In  doing  so,  this  study  developed  two  surveys  (pre-  and  post-training)  using 
previously  validated  instrument  seales  to  obtain  measurements  that  may  be  indieative  of 
training  effeetiveness.  In  addition,  a  proposed  researeh  model  was  tested  and  found 
insuffioient  in  identifying  appropriate  relationships  for  the  measurements  obtained. 
Furthermore,  a  revised  model  was  proposed  whieh  appropriately  identified  the 
relationships  between  the  variables  retained  and  was  subsequently  supported  by  the 
measurements  obtained.  Finally,  regression  analysis  was  used  to  proposed  mathematieal 
equations  for  the  prediction  of  training  outcomes  in  the  revised  model. 

Three  classes  of  BCOT  students  were  surveyed  with  a  paper-and-pencil 
instrument  that  resulted  in  thirty-nine  usable  responses.  From  these  responses, 
correlation  analysis  was  conducted  between  the  training  effectiveness  variables  identified 
from  the  literature.  ANOVA  was  conducted  to  examine  the  mean  change  between  the 
pre-  and  post-training  characteristics  of  affective  organization  commitment  and  task- 
related  self-efficacy.  Regression  analysis  was  used  to  further  the  study  with  predictive 
models.  This  chapter  presents  conclusions,  implications  for  the  Air  Force,  implications 
for  the  researcher,  limitations  of  the  study,  and  recommendations  for  future  research 
based  on  the  analysis  of  the  data. 
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Discussion 


Research  question  one,  “Based  on  the  literature,  which  constructs  are  appropriate 
for  measuring  training  effectiveness?”,  was  answered  after  an  extensive  review  of  the 
training  evaluation  literature.  Based  on  that  review,  trainee  demographics  (viz.,  age, 
gender,  MAJCOM  assignment,  time  in  service,  and  previous  communications 
experience),  trainee  characteristics  (viz.,  affective  organization  commitment,  task-related 
self-efficacy,  learning  self-efficacy,  and  motivation  to  learn),  and  organization 
characteristics  (viz.,  organization  support,  organization  constraints,  and  opportunity  to 
perform)  were  selected  to  examine  relationships  with  training  outcomes  (viz.,  training 
performance,  training  reactions,  and  motivation  to  transfer).  Support  for  each  variable 
selected  was  presented  in  chapter  two.  After  identifying  the  variables,  a  research  model 
(Figure  2)  was  proposed  to  study  the  identified  trainee  and  organization  characteristics 
and  their  influence  on  certain  training  outcomes  used  to  measure  training  effectiveness. 

Research  question  two,  “What  is  the  relationship  between  trainee  characteristics 
and  training  outcomes?”,  was  addressed  with  hypotheses  one,  three,  four,  and  five. 
Hypothesis  one  was  posited  to  examine  the  relationships  between  pre-training  trainee 
characteristics  and  training  outcomes.  Hypothesis  three  was  posited  to  examine 
relationships  between  trainee  demographics  and  trainee  characteristics.  Hypothesis  four 
was  posited  to  examine  the  relationship  between  pre-  and  post-training  trainee 
characteristics.  And  hypothesis  five  was  posited  to  examine  relationships  between 
training  outcomes  and  post-training  trainee  characteristics. 

Hypothesis  one  addressed  the  relationships  between  pre-training  trainee 
characteristics  and  training  outcomes.  All  of  the  sub-hypotheses  predicted  positive 
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relationships  between  eaeh  trainee  eharaeteristic  and  eaeh  training  outeome.  In  the 
original  proposed  researeh  model,  one-third  of  the  twelve  sub-hypotheses  were 
supported.  Training  outeomes,  training  reaetion-total  (TR-TTL)  and  motivation  to 
transfer  (MTT),  both  had  significant  correlations  with  affective  organization  commitment 
(AOC)  and  training  motivation  (TM).  Performance  did  not  significantly  correlate  with 
any  of  the  trainee  characteristics.  In  the  revised  model,  the  expected  correlations  between 
TM  and  learning  self-efficacy;  TM  and  TR-TTL;  and  TM  and  MTT  were  all  supported. 
Furthermore,  TM  and  the  proposed  training  effectiveness  variable  showed  a  significant 
positive  correlation.  These  results  indicate  that,  as  supported  by  previous  studies, 
training  motivation  is  positively  correlated  with  learning  self-efficacy  and  training 
outcomes.  Thus  training  motivation  contributes  positively  to  training  effectiveness  and 
should  be  considered  in  models  studying  training  effectiveness. 

Hypothesis  three  addressed  the  relationships  between  pre-training  trainee 
characteristics  and  trainee  demographics.  All  of  the  sub-hypotheses  predicted  positive 
relationships  between  each  trainee  characteristic  and  each  trainee  demographic  except  for 
gender  and  MAJCOM  assignment.  Relationships  between  gender  and  MAJCOM 
assignment  and  each  trainee  characteristic  were  suspected,  but  were  not  posited.  In  the 
original  proposed  research  model,  only  two  of  the  twenty  sub-hypotheses  were  supported. 
Age  was  positively  correlated  with  learning  self-efficacy  (LSE).  This  was  an  expected 
outcome  given  that  older  trainees  should  have  more  confidence  in  their  learning  ability. 
MAJCOM  assignment  also  positively  correlated  with  LSE.  Given  the  poor  results  for 
correlations  between  trainee  characteristics  and  demographics,  examination  of 
demographic  information  was  not  retained  for  the  revised  model.  Eurther  support  for 
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eliminating  demographic  information  from  the  model  was  revealed  with  additional 
examination  of  the  correlation  matrix.  Of  the  six  other  significant  demographics 
correlations,  three  were  within  the  demographics  characteristics  and  expected.  Therefore, 
for  this  sample,  demographics  did  not  appear  to  have  a  significant  role  in  determining 
training  effectiveness. 

Hypothesis  four  addressed  the  relationships  between  pre-training  trainee 
characteristics  and  post-training  trainee  characteristics  (viz.,  affective  organization 
commitment  -  AOC  and  task-related  self-efficacy  -  TRSE).  Correlation  analysis  and 
ANOVA  were  used  to  examine  the  data  collected.  Both  were  expected  to  increase  from 
the  pre-training  measure  to  the  post-training  measure.  However,  only  TRSE  showed  a 
significant  increase  over  time.  This  may  be  indicative  of  learning  during  the  course 
which  is  a  desired  result  of  training.  Both  pre-  and  post-training  measures  of  TRSE  were 
retained  in  the  revised  model.  Pre-  and  post-training  AOC  measurements  did  not  show  a 
statistically  significant  change  and  resulted  in  the  measure  being  retained  for  the  revised 
model,  but  only  as  a  post-training  measure  because  of  positive  correlations  with  training 
outcomes.  Pre-  and  post-training  TRSE  measurement  should  be  retained  for  the  purpose 
of  measuring  immediate  learning  of  the  training  course. 

Hypothesis  five  addressed  the  relationships  between  post-training  trainee 
characteristics  and  training  outcomes.  All  of  the  sub-hypotheses  predicted  positive 
relationships  between  each  trainee  characteristic  and  each  training  outcome.  In  the 
original  proposed  research  model,  four  of  the  six  sub-hypotheses  were  supported. 

Training  outcomes  (training  reaction-total  -  TR-TTL  and  motivation  to  transfer  -MTT) 
both  had  significant  correlations  with  affective  organization  commitment  (AOC)  and 
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task-related  self-effieacy  (TRSE).  Performanee  did  not  significantly  correlate  with  either 
of  the  trainee  characteristics  and  was  not  retained  in  the  revised  model.  In  the  revised 
model,  the  expected  correlations  between  training  characteristics  and  training  outcomes 
were  all  supported.  Furthermore,  AOC  and  TRSE  and  the  proposed  training 
effectiveness  variable  showed  significant  positive  correlations.  These  results  indicate 
that  AOC  and  TRSE  also  have  positive  effects  on  training  outcomes.  Thus  AOC  and 
TRSE  should  be  considered  when  studying  training  effectiveness. 

Research  question  three,  “What  is  the  relationship  between  organization 
characteristics  and  training  outcomes?”,  was  addressed  directly  by  hypothesis  two. 
Hypothesis  two  addressed  the  relationships  between  organization  characteristics  and 
training  outcomes.  All  of  the  sub-hypotheses  predicted  positive  relationships  between 
each  organization  characteristic  and  each  training  outcome,  except  for  organization 
constraints.  Organization  constraints  (OC)  were  predicted  to  correlate  negatively  with 
training  outcomes.  In  the  original  proposed  research  model,  four  of  the  nine  sub¬ 
hypotheses  were  supported.  Training  outcomes  (training  reaction-total  -  TR-TTL  and 
motivation  to  transfer  -MTT)  both  had  significant  correlations  with  organization  support 
(OS)  and  opportunity  to  perform  (OTP).  Performance  did  not  significantly  correlate  with 
any  of  the  organization  characteristics  and  was  not  retained  in  the  revised  model.  OC  did 
not  have  significant  negative  correlations  as  expected,  but  because  of  the  significant 
correlations  with  the  other  two  organization  characteristics,  was  retained  as  a  possible 
moderating  variable.  In  the  revised  model,  the  expected  correlations  between 
organization  characteristics  and  training  outcomes  were  all  supported.  Furthermore,  OS 
and  OTP  and  the  proposed  training  effectiveness  variable  showed  significant  positive 
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correlations.  These  results  indieate  that  OS  and  OTP  may  have  positive  effeets  on 
training  outeomes.  Thus  OS  and  OTP  (and  OC  as  a  possible  moderating  variable)  should 
be  considered  when  studying  training  effectiveness. 

Furthermore,  regression  analysis  for  the  revised  model  was  performed  to  provide 
mathematical  models  for  predietion  of  training  outcomes,  training  reaetion-total  and 
motivation  to  transfer,  and  the  new  training  effeetiveness  variable.  Testing  of  the 
mathematieal  models  for  their  predietive  eapabilities  will  be  discussed  in  the  future 
researeh  section. 

Implications  for  the  Air  Force 

This  study  demonstrated  the  importanee  between  trainee  and  organization 
eharaeteristics  and  training  outeomes.  Findings  indieated  training  motivation,  moderated 
by  learning  self-effleaey,  task-related  self-effieaey,  and  affeetive  organization 
eommitment  are  important  trainee  eharaeteristics  for  training  effectiveness  measurement. 
Also,  organization  support  and  opportunity  to  perform,  possibly  moderated  by 
organization  eonstraints,  are  important  organization  oharaeteristies  for  training 
effectiveness  measurement.  Furthermore,  training  reaetion  and  motivation  to  transfer  are 
pertinent  training  outeomes  and  should  be  studied  as  training  effeetiveness  measures. 
Subsequently,  a  more  streamlined  training  effeetiveness  model  was  appropriate  and 
presented  for  future  researeh. 

Identifying  these  important  eharaeteristics  as  influential  to  training  effectiveness 
may  help  supervisors  and  leadership  in  positively  affeeting  trainees  before,  during,  and 
after  training  eourses.  Indications  that  performance  scores  may  not  be  the  only  measure 
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of  training  effectiveness  are  prevalent  in  this  study.  While  performance  scores  are  the 
most  common  way  the  Air  Force  measures  training  performance,  identifying  other 
influential  characteristics  may  increase  overall  training  performance  as  well  as  post¬ 
training  performance.  Additionally,  affective  reactions  are  not  without  their  use.  As 
Alliger  et  al.  (1997)  suggested,  reactions  may  influence  political  decisions  such  as 
training  attendance,  training  funding  and  even  whether  training  is  offered  or 
discontinued.  Finally,  as  suggested  by  Miller  (1990),  even  if  training  has  no  measurable 
effect,  employees  gain  in  other  ways.  For  instance,  possible  affects  on  quality  of  life,  a 
highly  visible  Air  Force  concern,  may  occur  if  the  employee  feels  fulfilled,  more 
productive,  and  thus  happier  with  their  jobs  possibly  reaching  a  form  of  self-actualization 
on  Maslow’s  hierarchy. 

Implications  for  Researchers 

First,  this  study  proposed  a  research  model  integrating  previous  research  and 
hypothesized  discussions  to  measure  training  effectiveness.  After  initial  analyzation  of 
the  data  to  the  proposed  research  model,  a  revised  model  was  proposed  that  more 
effectively  captured  the  characteristics  proposed  to  measure  training  effectiveness.  As 
observed  by  Tracey  and  Tews  (1995),  training  does  not  occur  in  a  vacuum.  All  aspects  of 
the  trainee  and  the  trainee’s  environment  need  to  be  considered.  Effective  training 
depends  on  events  that  occur  before,  during,  and  after  training.  Factors  beyond  the 
traditional  view  of  effective  training  that  may  influence  the  extent  to  which  individuals 
ultimately  transfer  training  to  the  job  need  to  be  identified  and  investigated.  Validity  and 
reliability  of  the  revised  model  is  left  for  future  research. 
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In  addition,  the  study  proposed  a  new  training  effectiveness  variable,  a  product  of 
training  reaction  and  motivation  to  transfer,  that  positively  correlated  with  all  the  selected 
training  characteristics  in  the  revised  model.  Also,  implications  of  the  inability  to 
effectively  and  accurately  measure  the  constructs  identified  as  variables  of  training 
effectiveness  needs  to  be  addressed  (Miller,  1990).  Miller  suggests  that  even  with  the 
strictest  of  methodological  designs  and  the  best  possible  variable  control,  measurement  of 
such  constructs  may  not  be  possible.  Other  implications  may  include  not  only  what  to 
measure  but  when  to  measure  appropriate  training  effectiveness  variables.  Further 
research  in  order  to  increase  the  reliability  and  validity  of  measurement  scales,  as  well  as 
to  determine  the  appropriate  time  to  measure,  needs  to  continue  in  this  area  if  we  desire 
accuracy  in  measuring  training  effects  on  human  subjects. 

Limitations 

There  are  several  limitations  to  this  study.  First,  data  collected  was  self-reported 
by  the  trainees.  Self-report  data  hinge  on  the  accuracy  of  the  perceptions  of  the 
participants  in  the  training  program  (Sadri  &  Snyder,  1995).  Researchers  want  alpha 
change  which  is  caused  by  an  actual  change  in  the  construct  over  the  measurement 
period.  Beta  change  during  self-report  refers  to  the  measuring  instrument  being 
recalculated  by  the  participant  during  the  measurement  intervals.  Gamma  change  refers 
to  a  participant  redefining  some  relevant  information  at  the  post-test  measurement 
resulting  in  a  lower  self-assessment  and  while  change  may  have  occurred,  it  may  be  non¬ 
existent  or  minimized  at  the  post  measurement.  As  suggested  by  Miller  (1990),  an 
attempt  to  control  response  bias  can  be  made  by  using  special  instructions  to  participants 
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on  how  to  complete  the  survey.  This  study  attempted  to  eontrol  response-shift  bias  by 
providing  such  instructions.  However,  it  is  unclear  whether  there  was  a  response-shift 
bias  due  to  trainees’  reinterpretation  of  the  response  seale.  Additionally,  Noe  and 
Schmitt  (1986)  suggest  that  interviews  with  supervisors,  mentors,  and  peers  may 
strengthen  the  validity  of  the  self-report  information.  Due  to  time  limitations,  such 
interviews  were  unable  to  be  condueted. 

Next,  the  most  common  taxonomy  for  measuring  training  evaluation  is  by 
measuring  ehange  (Sackett  &  Mullen,  1993).  However,  Saekett  and  Mullen  propose 
other  more  effeetive  methods  to  measure  training  effeetiveness  using  pre-experimental 
design  as  opposed  to  quasi-experimental  design.  Design  is  dependant  on  the 
organizations  required  outcome.  Is  a  precise  change  in  level  of  knowledge,  skills,  and 
attitudes  required,  or  will  measurement  of  specific  level  of  achievement  be  enough?  If 
the  latter  is  the  ease,  a  pretest-posttest  no  eontrol  group  only  design  is  adequate.  Sackett 
and  Mullen  further  eontend  that  many  perceived  limitations  may  be  logically  ruled  out. 
For  example,  maturation  effects  due  to  short  time  duration  of  a  training  program  may  be 
ruled  out.  Therefore,  Sackett  and  Mullen’s  argument  for  use  of  pre-experimental  design, 
paired  with  careful  investigation  of  the  plausibility  of  various  threats,  is  better  than  no 
evaluation  due  to  design  limitations.  In  order  to  determine  if  support  for  the  hypotheses 
identified  in  the  previous  chapter  existed,  a  pre-experimental  design  commonly  used  for 
training  evaluation  was  developed.  According  to  Tannenbaum  and  Woods  (1992),  the 
ease  study  is  used  to  examine  one  set  of  trainees  without  comparing  them  to  any  other 
group.  Collection  of  pre-  and  post-training  measures  at  several  points  may  establish  a 
clearer  baseline  for  eomparison  when  time  permits.  Researeh  design  to  measure  behavior 
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change  in  which  no  partitioning  is  possible  is  further  supported  by  Tannenbaum  and 
Woods’  study.  A  single,  trained  group  measurement  can  provide  a  dependent  training 
effect  estimate  (Alliger  &  Tannenbaum,  1996). 

Additionally,  small  sample  size  was  a  limitation.  As  the  magnitude  of  the 
evaluation  increases,  ambiguity  of  interpretation  usually  decreases  (Tannenbaum  & 
Woods,  1992).  However,  Sackett  and  Mullen  (1993)  present  the  trade-offs  between 
internal  validity  and  statistical  conclusion  validity  when  a  large  sample  size  is  not 
available.  Acknowledging  that  small  sample  sizes  usually  have  low  statistical  power, 
they  argue  that  statistical  power  is  higher  with  a  pre-experimental  design  and  elimination 
of  the  control  group.  Further,  because  sample  size  may  be  limited  by  certain  money  and 
time  constraints  as  was  the  case  in  this  study,  it  may  be  reasonable  to  trade-off  internal 
validity  for  statistical  conclusion  validity.  They  conclude  that  creation  of  a  control  group 
to  achieve  greater  internal  validity  may  extract  too  great  a  price  in  terms  of  threats  to 
statistical  conclusion  validity. 

Finally,  Kirkpatrick  (1977)  stresses  the  point  that  proof  of  training  effectiveness 
comes  from  eliminating  all  other  factors  that  could  cause  outcomes  perceived  to  come 
from  training.  While  this  may  seem  the  case  with  this  training  because  of  the  course 
length  and  given  the  fact  that  the  only  requirement  during  this  period  was  for  the  trainees 
to  attend  the  BCOT  course,  it  cannot  be  ruled  out  that  other  confounding  variables  that 
were  not  measured  may  have  affected  training  outcomes.  For  instance,  this  course  was 
slightly  shortened  to  twelve  weeks  in  some  instances  to  graduate  students  prior  to  the 
holiday  season.  Trainees  were  not  asked  about  their  perception  about  whether  they 
received  quality  training  given  this  fact.  Thus,  affects  from  a  possible  negative  impact 
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due  to  the  shortened  eourse  was  not  eaptured.  Also,  quality  of  life  issues  were  not 
evaluated.  Some  trainees  mentioned  issues  with  their  living  quarters  that  may  have 
negatively  affeeted  their  performanee.  Again,  affects  from  this  possible  negative  impact 
were  not  measured,  either.  It  may  be  appropriate  to  include  these  two  areas  as  well  as 
others  in  future  research. 

Future  Research 

There  are  several  opportunities  for  future  research  in  this  area.  First,  validation  of 
the  revised  research  model  may  provide  support  for  the  training  effectiveness  variable 
proposed  in  this  study.  Additionally,  generalizability  for  the  model  may  be  supported  by 
using  other  occupational  training  courses  in  military  and  civilian  settings.  With  such 
diversity,  further  understanding  of  organization  constraints,  organization  support,  and 
opportunity  to  perform  and  their  affect  on  training  outcomes  may  develop  (Peters  et  ah, 
1988).  It  may  also  be  beneficial  to  use  this  model  as  a  first  step  in  the  development  of 
longitudinal  approaches  to  training  evaluation  to  go  beyond  the  traditional  pre-post 
designs  (Alliger  &  Tannenbaum,  1996).  Proper  planning  to  follow  a  trainee’s  progress 
before,  during,  and  after  training  will  be  required  to  collect  the  multiple  samples  required 
to  identify  appropriate  trends. 

Next,  research  concerning  influences  on  training  effectiveness  beyond  the  focus 
of  the  training  program  and  its  attributes  along  with  adopting  a  more  global  or  systems 
perspective  is  needed  (Tannenbaum  &  Yukl,  1992).  Research  to  measure  the  benefits 
that  accrue  after  training  has  been  completed  as  well  as  over  the  life  of  the  trainee’s 
career  are  supported  in  the  literature  (Parry,  1996).  Further,  in  the  Air  Force,  there  is  a 
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shifting  paradigm  of  how  and  when  to  provide  an  officer  with  training  for  professional 
development.  With  the  realization  that  training  and  education  throughout  an  officer’s 
career  is  required,  continuous  training  evaluation  is  needed  in  order  to  ensure  appropriate 
training  is  provided.  Therefore,  knowing  what  to  measure,  and  in  what  context,  is  also 
important. 

In  addition,  through  regression  analysis,  three  mathematical  equation  models  were 
developed  to  use  as  prediction  models  for  training  outcomes  and  the  training 
effectiveness  variable.  Support  and  validation  of  these  models  is  needed  in  future 
studies.  If  these  models  may  be  used  to  predict  training  outcomes  and  training 
effectiveness,  further  variable  reduction  would  enhance  training  evaluation  measurement 
and  produce  a  more  efficient  training  effectiveness  model. 

Finally,  knowing  when  to  measure  the  variables  concerned  with  training 
effectiveness  is  needed.  Bell  and  Kerr  (1987)  note  that  behavior  change  should  be 
measured  from  3  to  23  months  after  completion  of  the  training  program.  Alliger  et  al. 
(1997)  note  that  utility  reaction  measurement  may  be  better  measured  at  times  other  than 
immediately  following  training.  Research  in  this  area  may  provide  insight  on  the  best 
time  to  measure  the  training  effectiveness  characteristics  posited  in  this  study. 

Summary 

Lack  of  training  evaluation  due  to  misconceptions  may  be  more  harmful  than 
suspected.  In  the  military  setting,  a  captive  audience  for  measurement  is  presented  and 
offers  the  perfect  opportunity  to  study  the  training  effectiveness  construct.  Not  knowing 
whether  training  is  effective  is  wasteful  not  only  to  the  trainee,  but  to  the  organization  for 
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which  the  trainee  is  expected  to  work.  Identifying  variables  that  may  be  indicative  of 
training  effectiveness  other  than  a  quantitative  performance  seore  may  prove  to  provide 
the  most  benefits  from  training  in  that  supervisors  and  leadership  will  be  able  to  have 
more  influence  over  officers  (in  the  military  setting)  before,  during,  and  after  training. 
Therefore,  having  the  eapability  to  increase  training  effectiveness  by  understanding  what 
to  influence  would  benefit  any  organization. 

Results  of  this  study  suggest  that  certain  trainee  eharaeteristies,  organization 
charaeteristics,  and  training  outeomes  are  pertinent  to  training  effectiveness.  On  first 
review,  several  variables  from  the  original  proposed  research  model  could  be  eliminated. 
A  revised  model  was  presented  that  more  completely  supported  the  characteristics 
identified  as  influential  to  training  effectiveness.  Identifying  these  influences  on  training 
effectiveness  may  prove  to  be  benefieial  in  that  organizations  may  be  able  to  positively 
affect  trainees  before,  during,  and  after  training,  thus  increasing  training  effectiveness. 
Finally,  immediate  training  outcomes  may  be  more  indicative  of  future  performanee  than 
was  previously  thought  and  deserves  future  research. 
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Appendix  A 


A  Study  of  Training  Effectiveness 


There  has  been  widespread  support  for  the  positive  effeets  in  training  effectiveness  identified  by 
measuring  trainee  and  organization  characteristics  in  civilian  and  military  groups.  Within 
military  organizations,  research  findings  have  lent  support  to  the  importance  of  measuring 
training  effectiveness  to  determine  certain  training  courses  are  meeting  their  goals.  In  addition  to 
increased  performance  there  is  evidence  that  immediate  measures  of  training  effectiveness  may 
be  indicative  of  the  level  of  training  skills  that  transfer  to  the  work  environment. 

Considering  the  body  of  evidence  that  touts  the  influence  of  trainee  and  organization 
characteristics  on  training  effectiveness,  it  is  not  surprising  that  military  leaders  are  interested  in 
the  evaluation  of  training  effectiveness  to  determine  whether  training  provided  is  appropriate  and 
influential  on  trainees  and  their  ability  to  transfer  skills  learned  to  the  work  environment.  With 
this  in  mind,  the  Basic  Communications  Officer  Training  (BCOT)  course  was  designed  to  provide 
training  to  new  Communications  and  Information  Officers  to  ensure  they  have  a  solid  foundation 
in  communications  and  information  skills  to  be  successful  in  the  United  States  Air  Force. 

However,  there  is  still  some  question  about  what  to  measure  to  determine  training  effectiveness. 
This  research  will  test  a  proposed  model  to  measure  trainee  and  organization  characteristics  that 
may  have  predicative  capability  in  determining  training  effectiveness  of  the  BCOT  course. 


Capt.  Grace  M.  Beck 
AFIT/ENV  BLDG  640  Box  4019 

2950  Hobson  Way 

Wright-Patterson  AFB  OH  45433-7765 
Email:  grace.beck(S)afit.edu 

Phone:  DSN  785-3636  ext.  6019,  commercial  (937)  255-3636  ext.  6019 
Fax:  DSN  986-4699;  commercial  (937)  656-4699 


Please  remove  this  page  and  retain  for  your  record 
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Privacy  Notice 


The  following  information  is  provided  as  required  by  the  Privacy  Act  of  1974: 

Purpose:  To  obtain  information  regarding  training  effectiveness  of  BCOT. 

Routine  Use:  The  survey  results  will  be  used  to  provide  developmental  feedback  for  Training  programs 
within  the  Air  Force.  A  final  report  will  be  provided  to  participating  organizations.  No  analysis  of 
individual  responses  will  be  conducted  and  only  members  of  the  Air  Force  Institute  of  Technology  research 
team  will  be  permitted  access  to  the  raw  data. 

Participation:  Participation  is  VOLUNTARY.  No  adverse  action  will  be  taken  against  any  member  who 
does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  the  survey. 

Anonymity:  We  would  greatly  appreciate  your  participation.  Your  input  is  important  for  us  to  completely 
understand  your  trainee  and  organization  characteristics.  ALL  ANSWERS  ARE  STRICTLY 
AN  ON  YMOU  S.  Thus,  your  name  will  not  be  included  anywhere  on  this  questionnaire. 


INSTRUCTIONS 

•  Base  your  answers  on  your  own  thoughts  &  experiences 

•  Please  print  your  answers  clearly  when  providing  comments 

•  Make  dark  marks  when  asked  to  use  specific  response  options  (feel  free  to  use  an  ink  pen) 

•  Avoid  stray  marks  and  if  you  make  corrections  erase  marks  completely  or  clearly  indicate  the 
errant  response  if  you  use  an  ink  pen 


MARKING  EXAMPLES 


Right  Wrong 

•  ®  0  © 


IDENTIFICATION 

As  part  of  this  study,  we  will  need  to  match  your  responses  to  the  pre-training  and  post-training 
surveys,  as  well  as  to  the  final  course  grade  you  achieve.  In  order  to  do  this,  we  will  be  asking  for 
part  of  your  social  security  number.  At  the  end  of  the  data  collection  period,  this  information  will 
be  discarded  from  the  data  to  ensure  the  promised  anonymity. 


PLEASE  ENTER  THE  LAST  5  DIGITS  OF  YOUR  SOCIAL  SECURITY 
NUMBER  BELOW: 
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Section  I 

PRE-TRAINING  SURVEY 


This  section  asks  questions  concerning  commitment,  self-efficacy  and  motivation. 
For  each  statement,  please  fill  in  the  circle  for  the  number  that  indicates  the  extent 
to  which  you  agree  the  statement  is  true.  Use  the  scale  helow  for  your  responses. 


®  @  (D  @  ©  ® 

strongly  Disagree  Slightly  Neither  Agree  Slightly  Agree  Agree 

Disagree  Disagree  nor  Disagree 

© 

Strongly 

Agree 

1 .  When  making  my  best  effort,  I  would  be  able  to  discuss 
Communication  and  Information  (Cl)  authority  and  responsibility  from 
the  MAJCOM  level  down  to  the  Cl  Systems  Officer  at  the  base  level, 
including  deployed  operations  and  expeditionary  aerospace  forces. 

®  ©  ©  @ 

©  ©  © 

2.  The  Air  Force  has  a  great  deal  of  personal  meaning  to  me. 

©  ©  ©  © 

®  ©  © 

3.  This  training  is  really  a  waste  of  time. 

©  ©  ©  @ 

©  ©  © 

4.  I  think  that  I  could  easily  become  as  attached  to  another 
organization  as  I  am  to  the  Air  Force. 

©  ©  ©  @ 

®  ©  © 

5.  When  making  my  best  effort,  I  would  be  able  to  explain  how  to  treat 
Air  Force  communications  systems  as  weapons  systems. 

©  ©  ©  @ 

©  ©  © 

6.  Generally,  I  am  enthusiastic  about  learning  new  things. 

©  ©  ©  © 

©  ©  © 

7.  I'm  certain  I  can  understand  the  most  difficult  material  presented  in 
the  readings  for  this  course. 

©  ©  ©  © 

©  ©  © 

8.  I  would  be  very  happy  to  fulfill  a  career  in  the  Air  Force. 

©  ©  ©  © 

©  ©  © 

9.  When  making  my  best  effort,  I  would  be  able  to  confidently  install, 
configure,  and  manage  the  Windows  NT  operating  system. 

©  ©  ©  © 

©  ©  © 

10.  I  enjoy  discussing  the  Air  Force  with  those  not  in  the  Air  Force. 

©  ©  ©  © 

©  ©  © 

1 1 .  I'm  confident  I  can  understand  the  basic  concepts  taught  in  this 
course. 

©  ©  ©  © 

©  ©  © 

12.  I  expect  to  do  well  in  this  training. 

©  ©  ©  © 

©  ©  © 

13.  When  making  my  best  effort,  I  would  be  able  to  plan  and  deploy  a 
communications  network. 

©  ©  ©  © 

©  ©  © 

14.  I  do  not  feel  like  ‘part  of  the  Air  Force  family’. 

©  ©  ©  © 

©  ©  © 

15.  I'm  certain  I  can  master  the  skills  being  taught  in  this  course. 

©  ©  ©  © 

©  ©  © 

16.  When  making  my  best  effort,  I  would  be  able  to  configure  and 
manage  a  UNIX-based  network  operating  system. 

©  ©  ©  © 

©  ©  © 
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strongly  Disagree  Slightly  Neither  Agree  Slightly  Agree  Agree  Strongly 

Disagree _ Disagree  nor  Disagree _ Agree 


17.  I  expect  that  this  training  will  help  me  a  lot  in  the  future. 

®  ©  (D  @  (D  ®  @ 

18.  Considering  the  difficulty  of  the  blocks,  the  instructors,  and  my 
skills,  I  think  I  will  do  well  in  this  course. 

®  ©  ©  @  ®  ®  ® 

19.  I  really  feel  as  if  the  Air  Force's  problems  are  my  own. 

®  ©  ®  @  ®  ®  © 

20.  I  am  keen  to  learn  more  about  the  subjects  covered  in  this  training. 

®  ©  ®  @  ®  ®  ® 

21 .  When  making  my  best  effort,  I  would  be  able  to  discuss  the  facets 
of  life  cycle  management  from  a  communications  perspective. 

®  ©  ®  @  ®  ®  ® 

22.  I  do  not  feel  a  strong  sense  of  belonging  to  the  Air  Force. 

®  ©  ®  @  ®  ®  © 

23.  When  making  my  best  effort,  I  would  be  able  to  successfully 
manage  an  Air  Force  communications  project. 

®  ©  ®  @  ®  ®  © 

24.  I'm  confident  I  can  do  an  excellent  job  on  the  assignments  and  tests 
in  this  training  course. 

®  ©  ®  @  ®  ®  © 

25.  When  making  my  best  effort,  I  would  be  able  to  identify  the  role  of 
Cl  officers  and  civilian  professionals  within  the  framework  of  the  Air 
Force  Mission. 

®  ©  ®  @  ®  ®  © 

26.  I  do  not  feel  'emotionally  attached'  to  the  Air  Force. 

®  ©  ®  @  ®  ®  © 

27.  When  making  my  best  effort,  I  would  be  able  to  identify  current 
communications  systems  employed  to  support  the  US  Air  Force 
mission. 

®  ©  ®  @  ®  ®  © 

28.  Generally,  I  prefer  to  keep  away  from  training  courses. 

®  ©  ®  @  ®  ®  © 

29.  When  making  my  best  effort,  I  would  be  able  to  discuss  the  facets 
of  network  operating  systems  and  the  roles  and  responsibilities  of  the 
Network  Control  Center  (NCC). 

®  ©  ®  @  ®  ®  © 

30.  I'm  confident  I  can  understand  the  most  complex  material 
presented  by  the  instructor  in  this  course. 

®  ©  ®  @  ®  ®  © 

3 1 .  When  making  my  best  effort,  I  would  be  able  to  describe  the 
Military  Satellite  Communication  systems  and  explain  how  commercial 
satellite  systems  impact  the  DOD. 

®  ©  ®  @  ®  ®  © 

32.  I  believe  I  will  receive  an  excellent  grade  in  this  course. 

®  ©  ®  @  ®  ®  © 

33.  Generally,  I  am  keen  to  take  up  any  learning  opportunity  offered  to 
me. 

®  ©  ®  @  ®  ®  © 

This  section  contains  items  regarding  your  personal  characteristics.  These  items  are 
very  important  for  statistical  purposes.  Respond  to  each  item  hy  WRITING  IN 
THE  INFORMATION  requested  or  CHECKING  THE  BOX  0  that  best  describes 
you. 

1  Age;  _ years 

2.  Gender: 

Male  Female 


3.  Rank: 


4.  MAJCOM  to  which  you  are  assigned; _ 

5.  Total  length  of  service  time  in  the  Air  Force  (include  prior  enlisted  time): 

_ years _ months 

6.  If  you  have  prior  enlisted  service,  were  you  in  the  communications  and  information 
field  prior  to  commissioning? 

Yes  No 


THANK  YOU  FOR  PARTICIPATING 
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COMMENTS  (OPTIONAL) 


For  future  development,  please  use  this  seetion  to  eomment  on  the  survey.  Feel  free  to 
include  any  additional  comments  you  have  concerning  this  research  or  your  training 
experience.  Again,  your  honest  and  frank  response  is  requested.  You  may  contact  me 
via  the  e-mail  and  phone  number  provided  on  the  cover  sheet.  Thanks  again  for  your 
participation. 


USE  THE  BACK  OE  THIS  SHEET  IE  ADDITIONAL  SPACE  IS  NEEDED 
Thank  You  for  your  Participation! 
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Appendix  B 


A  Study  of  Training  Effectiveness 


There  has  been  widespread  support  for  the  positive  effeets  in  training  effeetiveness  identified  by 
measuring  trainee  and  organization  charaeteristics  in  civilian  and  military  groups.  Within 
military  organizations,  research  findings  have  lent  support  to  the  importance  of  measuring 
training  effectiveness  to  determine  certain  training  courses  are  meeting  their  goals.  In  addition  to 
increased  performance  there  is  evidence  that  immediate  measures  of  training  effectiveness  may 
be  indicative  of  the  level  of  training  skills  that  may  transfer  to  the  work  environment. 

Considering  the  body  of  evidence  that  touts  the  influence  of  trainee  and  organization 
characteristics  on  training  effectiveness,  it  is  not  surprising  that  military  leaders  are  interested  in 
the  evaluation  of  training  effectiveness  to  determine  whether  training  provided  is  appropriate  and 
influential  on  trainees  and  their  ability  to  transfer  skills  learned  to  the  work  environment.  With 
this  in  mind,  the  Basic  Communications  Officer  Training  (BCOT)  course  was  designed  to  provide 
training  to  new  Communications  and  Information  (Cl)  Officers  to  ensure  they  have  a  solid 
foundation  in  communications  and  information  skills  to  be  successful  in  the  United  States  Air 
Force. 


ffowever,  there  is  still  some  question  as  to  how  to  best  develop  training  effectiveness 
evaluations.  This  research  will  test  a  proposed  model  to  measure  trainee  and  organization 
characteristics  that  may  have  predicative  capability  in  determining  training  effectiveness 
of  the  BCOT  course. 


Capt.  Grace  M.  Beck 
AFIT/ENV  BLDG  640  Box  4019 

2950  Hobson  Way 

Wright-Patterson  AFB  OH  45433-7765 
Email:  grace.beck(a),afit.edu 

Phone:  DSN  785-3636  ext.  6019,  commercial  (937)  255-3636  ext.  6019 
Fax:  DSN  986-4699;  commercial  (937)  656-4699 


Please  remove  this  page  and  retain  for  your  record 
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Privacy  Notice 


The  following  information  is  provided  as  required  by  the  Privaey  Act  of  1974: 

Purpose:  To  obtain  information  regarding  training  effectiveness  of  BCOT. 

Routine  Use:  The  survey  results  will  be  used  to  provide  developmental  feedback  for 
Training  programs  within  the  Air  Force.  A  final  report  will  be  provided  to  participating 
organizations.  No  analysis  of  individual  responses  will  be  conducted  and  only  members 
of  the  Air  Force  Institute  of  Technology  research  team  will  be  permitted  access  to  the  raw 
data. 

Participation:  Participation  is  VOLUNTARY.  No  adverse  action  will  be  taken  against 
any  member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of 
the  survey. 

Anonymity:  We  would  greatly  appreciate  your  participation.  Your  input  is  important 
for  us  to  completely  understand  your  trainee  and  organization  characteristics.  ALL 
ANSWERS  ARE  STRICTLY  ANONYMOUS.  Thus,  your  name  will  not  be  included 
anywhere  on  this  questionnaire. 


INSTRUCTIONS 

•  Base  your  answers  on  your  own  thoughts  &  experiences 

•  Please  print  your  answers  clearly  when  providing  comments 

•  Make  dark  marks  when  asked  to  use  specific  response  options  (feel  free  to  use  an  ink  pen) 

•  Avoid  stray  marks  and  if  you  make  corrections  erase  marks  completely  or  clearly  indicate  the 

errant  response  if  you  use  an  ink  pen 

MARKING  EXAMPLES 

Right 

Wrong 

• 

(X)  (S  © 

IDENTIFICATION 

As  part  of  this  study,  we  will  need  to  match  your  responses  to  the  pre-training  and  post-training  surveys,  as 
well  as  to  the  final  course  grade  you  achieve.  In  order  to  do  this,  we  will  be  asking  for  part  of  your  social 
security  number. 

PLEASE  ENTER  THE  LAST  5  DIGITS  OF  YOUR  SOCIAL  SECURITY  NUMBER  BELOW: 
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POST-TRAINING  SURVEY 


i 

This  section  asks  questions  concerning  commitment,  self-efficacy,  training  reaction, 
motivation  to  transfer,  organization  support,  situational  constraints,  and  opportunity  to 
perform.  For  each  statement,  please  fill  in  the  circle  for  the  number  that  indicates  the 
extent  to  which  you  agree  the  statement  is  true.  Use  the  scale  below  for  your  responses. 


®  @  ®  @ 
strongly  Disagree  Slightly  Neither  Agree  nor 

Disagree  Disagree  Disagree 

Slightly  Agree 

©  ® 
Agree  Strongly 

Agree 

1 .  This  course  was  very  relevant  to  my  job. 

®  @  (D 

@  ®  ®  ® 

2.  My  supervisor  will  encourage  me  to  learn  new  skills  or  to  try  out 
new  ideas. 

®  (2)  (D 

@  ®  ®  ® 

3.  The  Air  Force  has  a  great  deal  of  personal  meaning  to  me. 

®  @  (D 

@  ®  ®  ® 

4.  I  will  have  the  opportunity  to  manage  an  Air  Force 
communications  project. 

®  @  (D 

@  ®  ®  ® 

5.  When  making  my  best  effort,  I  will  be  able  to  confidently  install, 
configure,  and  manage  the  Windows  NT  operating  system. 

®  @  (D 

@  ®  ®  ® 

6.  When  making  my  best  effort,  I  will  be  able  to  successfully  manage 
an  Air  Force  communications  project. 

®  (2)  (D 

@  ®  ®  ® 

7.  I  will  receive  adequate  information  from  other  sources  (e.g.,  co¬ 
workers,  departments,  outside  companies  or  agencies,  etc.)  needed 
to  perform  my  job  well. 

®  @  (D 

@  ®  ®  ® 

8.  I  really  feel  as  if  the  Air  Force's  problems  are  my  own. 

®  @  (D 

@  ®  ®  ® 

9.  This  course  was  of  great  practical  value  to  me  for  my  job. 

®  @  (D 

@  ®  ®  ® 

10.  When  making  my  best  effort,  I  will  be  able  to  identify  the  role  of 
Cl  officers  and  civilian  professionals  within  the  framework  of  the 
Air  Force  Mission. 

®  (2)  (D 

@  ®  ®  ® 

1 1 .  When  making  my  best  effort,  I  will  be  able  to  discuss  the  facets  of 
network  operating  systems  and  the  roles  and  responsibilities  of 
the  Network  Control  Center  (NCC). 

®  @  (D 

@  ®  ®  ® 

12.  I  will  have  the  opportunity  to  develop  or  use  Air  Force 
communications  systems  as  weapons  systems. 

®  (2)  (D 

@  ®  ®  ® 

13.  I  intend  to  use  what  I  have  learned  in  this  course. 

®  @  (D 

@  ®  ®  ® 

14.  I  found  this  course  difficult  to  understand. 

®  (2)  (D 

@  ®  ®  ® 
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®  @  ®  @  si*?ti 

Strongly  Disagree  Slightly  Neither  Agree 

Disagree  Disagree  nor  Disagree  ® 

Agree  Strongly  Agree 

15. 

When  making  my  best  effort,  I  will  be  able  to  identify  eurrent 
communications  systems  employed  to  support  the  US  Air  Force 
mission. 

®  (2)  (D  @  ©  ®  ® 

16. 

There  is  a  shortage  of  help  in  my  unit. 

® 

@  @  ®  ®  @ 

17. 

This  course  was  closely  related  to  my  job  needs. 

® 

©  ©  ©  ®  © 

© 

18. 

I  do  not  feel  'emotionally  attached'  to  the  Air  Force. 

® 

@  ©  @  ©  © 

19. 

The  physical  aspects  of  my  unit  (e.g.,  space,  lighting,  etc.)  will 
be  adequate. 

® 

@  ©  @  ©  © 

20. 

There  are  rewards  and  incentives  for  acquiring  and  using  new 
knowledge  and  skills  in  one's  job. 

® 

©  ©  @  ©  © 

® 

21. 

When  making  my  best  effort,  I  will  be  able  to  describe  the 
Military  Satellite  Communication  systems  and  explain  how 
commercial  satellite  systems  impact  the  DOD. 

® 

©  ©  @  ©  © 

® 

22. 

There  will  be  enough  time  available  to  complete  my  job  duties 
as  assigned. 

® 

©  ©  @  ©  © 

® 

23. 

I  will  have  to  opportunity  to  perform  life  cycle  management  on 
a  communications  system. 

® 

©  ©  @  ©  © 

® 

24. 

I  will  receive  sufficient  forewarning  to  plan  my  work  activities. 

® 

©  ©  @  ©  © 

® 

25. 

I  do  not  feel  like  'part  of  the  Air  Force  family' 

® 

©  ©  @  ©  © 

® 

26. 

My  co-workers  will  resist  new  ideas  or  the  use  of  new  work 
procedures. 

® 

©  ©  @  ©  © 

® 

27. 

I  will  have  the  opportunity  to  work  as  a  Cl  officer. 

® 

©  ©  ©  ©  © 

® 

28. 

I  will  have  the  opportunity  to  work  with  current 
communications  systems  employed  to  support  the  US  Air  Force 
mission. 

® 

©  ©  @  ©  © 

® 

29. 

I  do  not  feel  a  strong  sense  of  belonging  to  the  Air  Force. 

® 

©  ©  @  ©  © 

® 

30. 

Cl  officers  are  provided  with  resources  necessary  to  acquire  and 
use  new  knowledge  and  skills. 

® 

©  ©  @  ©  © 

® 
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®  (D  ®  @  © 

strongly  Disagree  Slightly  Neither  Agree  Agree 

Disagree  Disagree  nor  Disagree  ^ 

© 

Strongly 

Agree 

31. 

The  operating  budget  in  my  unit  is  suffieient  to  eover  the  amount 
of  work  produeed  in  my  unit. 

®  ©  ®  @  © 

© 

© 

32. 

I  will  have  suffieient  authority  to  eomplete  the  tasks  that  are 
assigned  to  me. 

®  ©  ©  @  ® 

© 

© 

33. 

I  feel  very  eommitted  to  applying  what  I  have  learned  in  this 
eourse  to  my  job. 

©  ©  ©  ©  ® 

® 

© 

34. 

This  eourse  was  very  good  fun. 

©  ©  ©  @  © 

® 

© 

35. 

I  will  have  to  opportunity  to  work  at  the  base  level  in  an 
aerospaee  expeditionary  foree. 

©  ©  ©  @  © 

® 

© 

36. 

Administrative  rules  or  polieies  will  hinder  my  effeetiveness  on 
the  job. 

©  ©  ©  @  © 

® 

© 

37. 

I  thought  this  eourse  was  a  tough  one. 

©  ©  ©  @  © 

® 

© 

38. 

When  making  my  best  effort,  I  will  be  able  to  explain  how  to 
treat  Air  Foree  eommunieations  systems  as  weapons  systems. 

©  ©  ©  @  © 

® 

© 

39. 

I  will  have  the  opportunity  to  work  with  network  operating 
systems  and  the  Network  Control  Center  (NCC). 

©  ©  ©  @  © 

® 

© 

40. 

My  unit  has  preseribed  ways  of  doing  things  that  must  be 
followed. 

©  ©  ©  @  © 

® 

© 

41. 

I  will  have  the  opportunity  to  eonfigure  and  manage  a  UNIX- 
based  network  operating  system. 

©  ©  ©  ©  © 

® 

© 

42. 

The  Air  Foree  offers  exeellent  training  programs. 

©  ©  ©  @  © 

® 

© 

43. 

My  job  duties  and  tasks  will  be  seheduled  in  an  effieient  manner. 

©  ©  ©  @  © 

© 

© 

44. 

I  will  have  the  opportunity  to  plan  and  deploy  a  eommunieations 
network. 

©  ©  ©  @  © 

© 

© 

45. 

There  is  a  performanee  appraisal  system  that  ties  rewards  to  the 
use  of  newly  aequired  knowledge  and  skills. 

©  ©  ©  @  © 

© 

© 

46. 

I  think  that  I  eould  easily  beeome  as  attaehed  to  another 
organization  as  I  am  to  the  Air  Foree. 

©  ©  ©  @  © 

© 

© 

47. 

I  will  have  adequate  supplies  (e.g.,  paper,  equipment  parts)  for 
performing  my  job. 

©  ©  ©  @  © 

© 

© 
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Q  @  ®  @  (D 

Strongly  Disagree  Slightly  Neither  Agree  Slightly  Agree 

Disagree  Disagree  nor  Disagree 

® 

Agree 

© 

Strongly  Agree 

48. 

I  really  enjoyed  this  course. 

© 

© 

© 

© 

© 

© 

49. 

I  will  have  to  opportunity  to  install,  configure,  and  manage  a 
Windows  NT  operating  system. 

® 

© 

@ 

© 

® 

50. 

When  making  my  best  effort,  I  will  be  able  to  configure  and 
manage  a  UNIX -base  network  operating  system. 

® 

© 

@ 

® 

© 

® 

51. 

I  found  this  course  very  hard  to  follow. 

® 

© 

© 

@ 

© 

© 

® 

52. 

When  making  my  best  effort,  I  will  be  able  to  discuss  the 
facets  of  life  cycle  management  from  a  communications 
perspective. 

® 

© 

© 

© 

® 

© 

® 

53. 

This  course  was  extremely  interesting. 

® 

© 

© 

© 

© 

© 

® 

54. 

The  Air  Force  rewards  Cl  officers  for  using  newly  acquired 
knowledge  and  skills  on  the  job. 

® 

© 

© 

© 

© 

© 

® 

55. 

When  making  my  best  effort,  I  will  be  able  to  discuss  Cl 
authority  and  responsibility  from  the  MAJCOM  level  down 
to  the  Cl  Systems  Officer  at  the  base  level,  including 
deployed  operations  and  expeditionary  aerospace  forces. 

® 

© 

© 

© 

© 

© 

® 

56. 

When  making  my  best  effort,  I  will  be  able  to  plan  and 
deploy  a  communications  network. 

® 

© 

© 

© 

© 

© 

® 

57. 

I  would  be  very  happy  to  fulfill  a  career  in  the  Air  Force. 

® 

© 

© 

© 

© 

© 

® 

58. 

I  will  have  the  opportunity  to  work  with  the  Military  Satellite 
Communication  system  and  commercial  satellite  systems. 

® 

© 

© 

© 

© 

© 

® 

59. 

I  enjoy  discussing  the  Air  Force  with  those  not  in  the  Air 
Force. 

® 

© 

© 

© 

© 

© 

® 

60. 

I  am  keen  to  apply  what  I  have  learned  in  this  course. 

® 

© 

© 

© 

© 

© 

® 

61. 

I  will  have  adequate  equipment  (e.g.,  computers,  software) 
for  performing  my  job. 

® 

© 

© 

© 

© 

© 

® 

62. 

Time  will  be  made  available  to  me  in  order  to  practice  new 
skills  or  to  experiment  with  different  work  procedures. 

® 

© 

© 

© 

© 

© 

® 

63. 

I  will  have  the  opportunity  to  receive  adequate  educational 
and/or  training  experiences  necessary  to  perform  my  job  well. 

® 

© 

© 

© 

© 

© 

® 

THANK  YOU  FOR  PARTICIPATING 
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COMMENTS  (OPTIONAL) 


For  future  development,  please  use  this  section  to  comment  on  the  survey.  Feel 
free  to  include  any  additional  comments  you  have  concerning  this  research  or  your 
training  experience.  Again,  your  honest  and  frank  response  is  requested.  If  you  have 
questions  or  would  like  a  response  to  a  comment,  please  include  your  e-mail  address  or 
phone  number.  Any  personal  information  you  provide  will  be  immediately  destroyed 
after  a  response  is  given  to  ensure  anonymity.  Thanks  again  for  your  participation. 


USE  THE  BACK  OE  THIS  SHEET  IE  ADDITIONAE  SPACE  IS  NEEDED 
Thank  You  for  your  Participation! 
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Appendix  C 


o 

K) 


_ Scale  Item  Summary _ 

Scale  Item _ Author  Reliability _ Original  Scale  Modified  Item _ 

by  Meyer  and  Allen  (1984;  7-point  Likert  Scale  from  1  (strongly  agree)  to  7  (strongly 
Affective  Commitment  Scale  Items  1990)  alpha  =  .87  disagree). 

Definition:  How  much  an  individual  identifies  with  and  is  involved  with  an  organization. 


1.  I  would  be  very  happy  to  spend  the  rest  of  my 
career  with  this  organization. 

2.  I  enjoy  discussing  my  organization  with  people 
outside  it. 


1 .  I  would  be  very  happy  to  fulfill  a  career  in 
the  Air  Force. 

2.  I  enjoy  discussing  the  Air  Force  with  those 
not  in  the  Air  Force. 


3.  I  really  feel  as  if  this  organization's  problems  are 
my  own. 

4.  I  think  that  I  could  easily  become  as  attached  to 
another  organization  as  I  am  to  this  one.  (R) 

5.  I  do  not  feel  like  'part  of  the  family'  at  my 
organization.  (R) 

6.  I  do  not  feel  'emotionally  attached'  to  this 
organization.  (R) 

7.  This  organization  has  a  great  deal  of  personal 
meaning  to  me. 

8.  I  do  not  feel  a  strong  sense  of  belonging  to  my 
organization.  (R) 


3.  I  really  feel  as  if  the  Air  Force's  problems 
are  my  own. 

4.  I  think  that  I  could  easily  become  as 
attached  to  another  organization  as  I  am  to  the 
Air  Force.  (R) 

5.  I  do  not  feel  like  'part  of  the  Air  Force 
family'.  (R) 

6.  I  do  not  feel  'emotionally  attached'  to  the  Air 
Force.  (R) 

7.  The  Air  Force  has  a  great  deal  of  personal 
meaning  to  me. 

8.  I  do  not  feel  a  strong  sense  of  belonging  to 
the  Air  Force.  (R) 


by  Robertson  and  Sadri 
(1993)  alpha  =  .97  (Version  A) 

Task-related  Self  Efficacy  and  .96  (Version  B)  How  well  performed  on  a  100-point  range 

Definition:  Trainees'  beliefs  in  their  ability  to  When  making  my  best  effort,  I  would  be  able  to 

perform  a  specific  task _ - _ 


Scale  Item 

Author  Reliability 

Original  Scale  Modified  Item 

1.  Foundations  for  the  Cl  Officer 

-  identify  the  role  of  Cl  officers  and  civilian 
professionals  within  the  framework  of  the  Air 
Force  Mission. 

2.  Communications  Fundamentals 

-  identify  current  communications  systems 
employed  to  support  the  US  Air  Force  mission. 

3.  Computer  Fundamentals 

-  discuss  the  facets  of  life  cycle  management 
from  a  communications  perspective. 

4.  Information  Assurance 

-  explain  how  to  treat  Air  Force 
communications  systems  as  weapons  systems. 

5.  Network  Fundamentals 

-  discuss  the  facets  of  network  operating 
systems  and  the  roles  and  responsibilities  of  the 
Network  Control  Center  (NCC). 

6.  Network  Operation  Systems  Lab 

-  configure  and  manage  a  UNIX-based  network 
operating  system. 

7.  Network  Applications  Lab 

-  confidently  install,  configure,  and  manage  the 
Windows  NT  operating  system. 

8.  Resource  Management 

-  successfully  manage  an  Air  Force 
communications  project. 

9.  Fixed  Communications 

-  discuss  Cl  authority  and  responsibility  from 
the  MAJCOM  level  down  to  the  Cl  Systems 
Officer  at  the  base  level,  including  deployed 
operations  and  expeditionary  aerospace  forces. 

10.  Space 

-  describe  the  Military  Satellite  Communication 
systems  and  explain  how  commercial  satellite 
systems  impact  the  DOD. 

1 1 .  Deployable  Communications 

-  plan  and  deploy  a  communications  network. 

Scale  Item _ Author  Reliability _ 

by  Pintrich  et  al.  (1993) 

Learning  Self-Efficacy  alpha  =  .93 

Definition:  Trainees'  beliefs  in  their  ability  to  learn  training  material 

1 . 1  believe  I  will  receive  an  excellent  grade  in  this 
class. 

2.  I'm  confident  I  can  do  an  excellent  job  on  the 
assignments  and  tests  in  this  course. 

3. 1  expect  to  do  well  in  this  class. 

4.  Considering  the  difficulty  of  this  course,  the 
teacher,  and  my  skills,  I  think  I  will  do  well  in  this 
class. 

5.  I'm  certain  I  can  understand  the  most  difficult 
material  presented  in  the  readings  for  this  course. 

6.  I'm  confident  I  can  understand  the  basic  concepts 

o  taught  in  this  course. 

7.  I'm  confident  I  can  understand  the  most  complex 
material  presented  by  the  instructor  in  this  course. 

8.  I'm  certain  I  can  master  the  skills  being  taught  in 
this  class. 

Motivation  to  Learn 

Definition:  Trainee’s  belief  in  their  motivation  to 
learn  in  general. 

1.  Generally,  I  am  enthusiastic  about  learning  new 
things 

2.  Generally,  I  prefer  to  keep  away  from  training 

courses  (reverse  scored) _ 


by  Warr,  Allen  &  Birdi 
(1999)  alpha  =  .76 


Original  Scale  Modified  Item _ 

7-point  Likert  scale  from  1  (not  at  all  trne  for  me)  to  7  (very 
trne  for  me). 

1 . 1  believe  I  will  receive  an  excellent  grade  in 
this  course. 

2.  I'm  confident  I  can  do  an  excellent  job  on  the 
assignments  and  tests  in  this  training  course. 

3. 1  expect  to  do  well  in  this  training. 

4.  Considering  the  difficulty  of  the  blocks,  the 
instructors,  and  my  skills,  I  think  I  will  do  well 
in  this  course. 

5.  I'm  certain  I  can  understand  the  most 
difficult  material  presented  in  the  readings  for 
this  course. 

6.  I'm  confident  I  can  understand  the  basic 
concepts  taught  in  this  course. 

7.  I'm  confident  I  can  understand  the  most 
complex  material  presented  by  the  instractor  in 
this  course. 

8.  I'm  certain  I  can  master  the  skills  being 
taught  in  this  course. 

5-point  Likert  scale  from  strongly  agree  to  strongly  disagree 


1 .  Generally,  I  am  enthusiastic  about  learning 
new  things. 

2.  Generally,  I  prefer  to  keep  away  from 

training  courses  (reverse  scored). _ 


Scale  Item 


Author  Reliability 


3.  Generally,  I  am  keen  to  take  up  any  learning 
opportunity  offered  to  me 

4.  I  am  keen  to  learn  more  about  the  subjeets  eovered 
in  this  eourse 

5.  I  expeet  that  this  eourse  will  help  me  a  lot  in  the 
future 

6.  This  eourse  is  really  a  waste  of  time  (reverse 
seored) 

by  Warr,  Allen  &  Bird! 

Training  Reaction  (1999) 

Definition:  Reaetion  to  the  training  eourse  by  three 
eategories:  enjoyment,  pereeived  usefulness,  and 
pereeived  diffieulty. 

Enjoymenf  alpha  =  .16 

1 .  I  really  enjoyed  this  eourse 

^  2.  This  eourse  was  very  good  fun 

o 

r-n  3,  This  eourse  was  extremely  interesting 

Pereeived  usefulness  alpha  =  .76 

1.  This  eourse  was  very  relevant  to  my  job 

2.  This  eourse  was  of  great  praetieal  value  to  me  for 
my  job 

3.  This  eourse  was  elosely  related  to  my  job  needs 

Pereeived  diffreultv  alpha  =  .16 

1,  I  found  this  eourse  very  hard  to  follow 

2,  I  thought  this  eourse  was  a  tough  one 

3,  I  found  this  eourse  diffieult  to  understand 


Motivation  to  Transfer 


by  Warr,  Allen  &  Birdi 
(1999)  alpha  =  ,79 


Original  Scale  Modified  Item _ 

3.  Generally,  I  am  keen  to  take  up  any  learning 
opportunity  offered  to  me. 

4.  I  am  keen  to  learn  more  about  the  subjeets 
eovered  in  this  training. 

5.  I  expeet  that  this  training  will  help  me  a  lot 
in  the  future. 

6.  This  training  is  really  a  waste  of  time 
(reverse  seored). 

5-point  Likert  scale  from  strongly  agree  to  strongly  disagree 


Enjoyment 

1.  I  really  enjoyed  this  eourse. 

2.  This  eourse  was  very  good  fun. 

3.  This  eourse  was  extremely  interesting. 
Pereeived  usefulness 

1 .  This  eourse  was  very  relevant  to  my  job. 

2.  This  eourse  was  of  great  praetieal  value  to 
me  for  my  job. 

3.  This  eourse  was  elosely  related  to  my  job 
needs. 

Pereeived  diffreultv 

1 .  I  found  this  eourse  very  hard  to  follow. 

2.  I  thought  this  eourse  was  a  tough  one. 

3.  I  found  this  eourse  diffieult  to  understand. 


5-point  Likert  scale  from  strongly  agree  to  strongly  disagree 


Scale  Item 

Author  Reliability 

Original  Scale  Modified  Item 

Definition:  The  trainee's  motivation  to  transfer  skills 
learned  in  the  training  eourse  to  the  job  environment 

1 .  I  am  keen  to  apply  what  I  have  learned  on  this 
eourse 

1 .  I  am  keen  to  apply  what  I  have  learned  in 
this  eourse. 

2.  I  intend  to  use  what  I  have  learned  on  this  eourse 

2.  I  intend  to  use  what  I  have  learned  in  this 

eourse. 

3.  I  feel  very  eommitted  to  applying  what  I  have 
learned  on  this  eourse  to  my  job 

3.  I  feel  very  eommitted  to  applying  what  I 
have  learned  in  this  eourse  to  my  job. 

Organization  Snpport  Items 

by  Tracey  et  al.  (2001) 
alpha  =  ..90 

5-point  Likert  scale  from  strongly  agree  to  strongly  disagree 

Definition:  Pereeived  polieies,  praetiees,  and 
proeedures  of  the  organization  that  demonstrates  the 
importanee  of  training  and  development  efforts. 

1.  There  is  a  performanee  appraisal  system  that  ties 
fmaneial  rewards  to  use  of  newly  aequired  knowledge 
and  skills. 

1 .  There  is  a  performanee  appraisal  system  that 
ties  rewards  to  the  use  of  newly  aequired 
knowledge  and  skills. 

2.  This  organization  offers  exeellent  training 
programs. 

2.  The  Air  Foree  offers  exeellent  training 
programs. 

3.  Employees  are  provided  with  resourees  neeessary 
to  aequire  and  use  new  knowledge  and  skills. 

3.  Cl  offieers  are  provided  with  resourees 
neeessary  to  aequire  and  use  new  knowledge 
and  skills. 

4.  There  are  rewards  and  ineentives  for  aequiring  and 
using  new  knowledge  and  skills  in  one's  job. 

4.  There  are  rewards  and  ineentives  for 
aequiring  and  using  new  knowledge  and  skills 
in  one's  job. 

5.  This  organization  rewards  employees  for  using 
newly  aequired  knowledge  and  skills  on  the  job. 

5.  The  Air  Foree  rewards  Cl  offieers  for  using 
newly  aequired  knowledge  and  skills  on  the 
job. 

Scale  Item 


Author  Reliability 


Original  Scale  Modified  Item 


o 


by  Mathieu  et  al.  (1992) 

(alpha  =  .85)  Note:  Only  14 

of  the  16  items  were  used  after  7-point  Likert  scale  from  "not  at  all"  to  "to  a  very  great 
Situational  Constraints  reviewing  initial  response.  extent" 

Definition:  Perceived  situations  in  the  work  setting 
that  may  limit  the  trainee  using  the  skills  learned  in 
the  course. 

To  what  extent : 


1 .  Do  you  receive  adequate  information  from  other 
sources  (e.g.,  co-workers,  departments,  outside 
companies  or  agencies,  etc.)  needed  to  perform  your 
job  well? 

2.  Do  you  have  adequate  equipment  (e.g., 
typewriters,  software)  for  performing  your  job? 

3.  Do  you  have  adequate  supplies  (e.g.,  paper, 
mailing  envelopes)  for  performing  your  job? 

4.  Is  there  a  shortage  of  help  in  your  office? 

5.  Have  you  had  the  opportunity  to  receive  adequate 
educational  and/or  training  experiences  necessary  to 
perform  your  job  well? 

6.  Is  there  enough  time  available  to  complete  your 
job  duties  as  assigned? 

7.  Are  the  physical  aspects  of  your  office  (e.g.,  space, 
lighting,  etc.)  adequate? 

8.  Are  your  job  duties  and  tasks  scheduled  in  an 
efficient  manner? 

9.  Do  you  have  sufficient  authority  to  complete  the 
tasks  that  are  assigned  to  you? 


1 .  I  will  receive  adequate  information  from 
other  sources  (e.g.,  co-workers,  departments, 
outside  companies  or  agencies,  etc.)  needed  to 
perform  my  job  well. 

2.  I  will  have  adequate  equipment  (e.g., 
computers,  software)  for  performing  my  job. 

3.  I  will  have  adequate  supplies  (e.g.,  paper, 
equipment  parts)  for  performing  my  job. 

4.  There  is  a  shortage  of  help  in  my  unit. 

5.  I  will  have  the  opportunity  to  receive 
adequate  educational  and/or  training 
experiences  necessary  to  perform  my  job  well. 

6.  There  will  be  enough  time  available  to 
complete  my  job  duties  as  assigned. 

7.  The  physical  aspects  of  my  unit  (e.g.,  space, 
lighting,  etc.)  will  be  adequate. 

8.  My  job  duties  and  tasks  will  be  scheduled  in 
an  efficient  manner. 

9.  I  will  have  sufficient  authority  to  complete 
the  tasks  that  are  assigned  to  me. 


Scale  Item 


Author  Reliability 


10.  Is  the  operating  budget  in  your  office  sufficient  to 
cover  the  amount  of  work  produced  in  your  unit? 

1 1 .  Do  administrative  rules  or  policies  hinder  your 
effectiveness  on  the  job? 

12.  Do  you  receive  sufficient  forewarning  to  plan 
your  work  activities? 

13.  Does  your  supervisor  encourage  you  to  learn  new 
skills  or  to  try  out  new  ideas? 

14.  Do  your  co-workers  resist  new  ideas  or  the  use  of 
new  work  procedures? 

15.  Does  your  office  have  prescribed  ways  of  doing 
things  that  must  be  followed? 

16.  Is  time  made  available  to  you  in  order  to  practice 
new  skills  or  to  experiment  with  different  work 
procedures? 

by  Ford  et  al.  (1992) 

Opportunity  to  Perform  alpha  =  ..74 

Definition:  Task  level  appraisal  of  perceived  trainee 
opportunity  to  perform  tasks  related  to 
skills/knowledge  learned  in  the  course. 

1 .  Foundations  for  the  Cl  Officer 

2.  Communications  Fundamentals 

3.  Computer  Fundamentals 

4.  Information  Assurance 


5.  Network  Fundamentals 


Original  Scale  Modified  Item _ 

10.  The  operating  budget  in  my  unit  is 
sufficient  to  cover  the  amount  of  work 
produced  in  my  unit. 

1 1 .  Administrative  rules  or  policies  will  hinder 
my  effectiveness  on  the  job. 

12.  I  will  receive  sufficient  forewarning  to  plan 
my  work  activities. 

13.  My  supervisor  will  encourage  me  to  learn 
new  skills  or  to  try  out  new  ideas. 

14.  My  co-workers  will  resist  new  ideas  or  the 
use  of  new  work  procedures. 

1 5 .  My  unit  has  prescribed  ways  of  doing 
things  that  must  be  followed. 

16.  Time  will  be  made  available  to  me  in  order 
to  practice  new  skills  or  to  experiment  with 
different  work  procedures. 

7-point  Likert  scale  from  strongly  agree  to  strongly  disagree 

I  will  have  the  opportunity  to  - 

-  work  as  a  Cl  officer. 

-  work  with  current  communications  systems 
employed  to  support  the  US  Air  Force  mission. 

-  perform  life  cycle  management  on  a 
communications  system. 

-  develop  or  use  Air  Force  communications 
systems  as  weapons  systems. 

-  work  with  network  operating  systems  and  the 

_ Network  Control  Center  (NCC). _ 


Scale  Item 

Author  Reliability 

Original  Scale  Modified  Item 

6.  Network  Operation  Systems  Lab 

-  eonfigure  and  manage  a  LTNIX-based  network 
operating  system. 

7.  Network  Applieations  Lab 

-  install,  eonfigure,  and  manage  a  Windows 

NT  operating  system. 

8.  Resouree  Management 

-  manage  an  Air  Foree  eommunieations  projeet. 

9.  Fixed  Communieations 

-  work  at  the  base  level  in  an  aerospaee 
expeditionary  foree. 

10.  Spaee 

1 1 .  Deployable  Communieations 

-  work  with  the  Military  Satellite 
Communieation  system  and  eommereial 
satellite  systems. 

-  plan  and  deploy  a  eommunieations  network. 
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